[Search for users] [Overall Top Noters] [List of all Conferences] [Download this site]

Conference rusure::math

Title:Mathematics at DEC
Moderator:RUSURE::EDP
Created:Mon Feb 03 1986
Last Modified:Fri Jun 06 1997
Last Successful Update:Fri Jun 06 1997
Number of topics:2083
Total number of notes:14613

792.0. "Power Regression" by COMET::ROBERTS (Peace .XOR. Freedom ?) Tue Nov 24 1987 17:06

    The "Lease Squares" solution for the linear regression formula
    
    ^
    y = m*x + k
    
    is the solution to the matrix equation
    
    | Sx� Sx |   | m |   | Sxy |
    |        | * |   | = |     |
    | Sx  n  |   | k |   | Sy  |
    
                                       n
    where "S" is an abbreviation for Sigma
                                      i=1
    
    
    I'm trying to derive the Least Squares equation for the power
    regression formula
    
    ^      b
    y = a*x
    
    but am having some difficulty.  I start with
    
                 b
    Z = S(y - a*x )�
    
                    b         2*b
    Z = S(y� - 2*a*x *y + a�*x   )
    
    I want to minimize Z with respect to a and b, so I take the partial
    derivatives and set them to zero:
    
                            b             2*b
    dZ/da = 0 implies -2*S(x *y) + 2*a*S(x   ) = 0
    
                                    b                    2*b
    dZ/db = 0 implies -2*a*S(ln(x)*x *y) + 2*a�*S(ln(x)*x   ) = 0
    
    after rearranging some terms and simplifying, I'm left with
    
         2*b       b
    a*S(x   ) = S(x *y)
    
               2*b             b
    a*S(ln(x)*x   ) = S(ln(x)*x *y)
    
    
    How can I get from here to the AX=B matrix equation that is so easily
    derived for the linear case?
T.RTitleUserPersonal
Name		DateLines
792.1can't get there from hereMATRIX::ROTHMay you live in interesting timesWed Nov 25 1987 06:1927
    It is only in the cases where the error is a (convex) quadratic form that
    the solution comes out with simple linear algebra.  For more general
    problems the solution involves nonlinear equations.  Moreover, you
    can't even guarantee that there is a minimum unless the sign of the
    determinant of the Hessian matrix (of second partial derivatives) is 
    checked.

    Also, such nonlinear equation fitting tends to be ill conditioned.
    Qualitatively b has much more influence than a when |x| >> 1, but much
    less when |x| << 1 - noise in the data would tend to send the solution
    all over the place...

    Try fitting against log(y) instead, which is back to fitting a linear
    equation.

	e(a,b) = log(y) - log(a) - b*log(x)
	e(c,b) = log(y) - c - b*log(x), c = log(a)

    Now,

	d e^2 / dc = -2*(log(y) - c - b*log(x))

	d e^2 / db = -2*(log(y) - c - b*log(x))*log(x)

    which is a set of linear equations...

    - Jim
792.2Easy As Falling Off A Log?COMET::ROBERTSPeace .XOR. Freedom ?Wed Nov 25 1987 10:5324
    I'd use the log curve fit, but it's necessary to use power curve.
    You did spur my imagination some, though.  What if I took the log
    of both sides?  Then, instead of
    
    ^      b
    y = a*x
    
    
    I'd have
    
       ^
    ln(y) = ln(a) + b*ln(x)
    
    which is linear.  My hasty calculations tell me that the system
    of equations would then be
    
    | n         S(ln(x))  |   | ln(a) |   | S(ln(y))       |
    |                     | * |       | = |                |
    | S(ln(x))  S(ln�(x)) |   | b     |   | S(ln(x)*ln(y)) |
    
    Is this on track?
    
    						/Dwayne
792.3CLT::GILBERTBuilderWed Nov 25 1987 15:503
                                              b
    That doesn't guarantee that Z = S((y - a*x )�) is minimized,
    but it should still give a good fit.
792.4log-logBLITZN::ROBERTSPeace .XOR. Freedom ?Wed Nov 25 1987 16:3623
>    < Note 792.3 by CLT::GILBERT "Builder" >
>
>
>                                              b
>    That doesn't guarantee that Z = S((y - a*x )�) is minimized,
>    but it should still give a good fit.

    Right.  Instead, Z would be such that
    
    Z = S((ln(y) - (ln(a)+b*ln(x))�)    is minimized.  Since this is the
    same as
    
                        b
    Z = S( ln�( y / (a*x ) ) ) 
    
    then instead of minimizing the sum of the squares of the differences,
    we're minimizing the sum of the squares of the natural logarithms of
    the quotients.  Interesting. I think if I display the data and
    regression curve on a graph, I'll use log-log paper.
    
    						/Dwayne