The Nature of Mathematical Modeling Neil Gershenfeld
PUBLISHED BY THE PRESS SYNDICATE OF THE UNIVERSITY OF CAMBRIDGE The Pitt Building, Trumpington Street, Cambridge CB2 1RP, United Kingdom CAMBRIDGE UNIVERSITY PRESS The Edinburgh Building, Cambridge CB2 2RU, UK http://www.cup.cam.ac.uk 40 West 20th Street, New York, NY 10011-4211, USA http://www.cup.org 10 Stamford Road, Oakleigh, Melbourne 3166, Australia c Cambridge University Press 1999 This book is in copyright. Subject to statutory exception and to the provisions of relevant collective licensing agreements, no reproduction of any part may take place without the written permission of Cambridge University Press. First published 1999 Printed in the United Kingdom at the University Press, Cambridge TypesetinMonotypeEhrhardt10 1 2 /13pt,inL ATEX2ε[EPC] A catalogue record of this book is available from the British Library Library of Congress Cataloguing in Publication data Gershenfeld, Neil A. The nature of mathematical modeling/ Neil Gershenfeld. p. cm. Includes bibliographical references and index. ISBN 0-521-57095-6 1. Mathematical models. I. Title. QA401.G47 1998 511.8 dc21 98-22029 CIP ISBN0521570956hardback
Contents Preface to the second edition Preface to the first edition page xiii xv 1 Introduction 1 1.1 SelectedReferences.......................... 3 Part One: Analytical Models 5 2 Linear Algebra 9 2.1 Numbers,Vectors,andMatrices.................... 9 2.2 SystemsofEquations.......................... 13 2.3 SingularValues............................. 16 2.4 FunctionsofMatrices......................... 18 2.5 SelectedReferences.......................... 20 2.6 Problems................................ 20 3 Ordinary Differential and Difference Equations 21 3.1 LinearDifferentialEquations..................... 21 3.2 SystemsofDifferentialEquationsandNormalModes......... 24 3.3 LaplaceTransforms.......................... 26 3.4 PerturbationExpansions........................ 30 3.5 DiscreteTimeEquations........................ 31 3.6 z-transforms.............................. 31 3.7 SelectedReferences.......................... 33 3.8 Problems................................ 34 4 Partial Differential Equations 36 4.1 TheOriginofPartialDifferentialEquations.............. 36 4.2 LinearPartialDifferentialEquations.................. 38 4.3 SeparationofVariables......................... 39 4.3.1 RectangularCoordinates.................... 40 4.3.2 CylindricalCoordinates.................... 41 4.3.3 SphericalCoordinates..................... 43 4.4 TransformTechniques......................... 44
DRAFT Contents vii 4.5 SelectedReferences.......................... 45 4.6 Problems................................ 45 5 Variational Principles 46 5.1 VariationalCalculus.......................... 46 5.1.1 Euler sequation........................ 46 5.1.2 IntegralsandMissingVariables................ 48 5.1.3 ConstraintsandLagrangeMultipliers............. 49 5.2 VariationalProblems.......................... 50 5.2.1 Optics:Fermat sprinciple................... 50 5.2.2 AnalyticalMechanics:Hamilton sprinciple.......... 50 5.2.3 Symmetry:Noether stheorem................ 51 5.3 RigidBodyMotion........................... 52 5.4 SelectedReferences.......................... 55 5.5 Problems................................ 55 6 Random Systems 56 6.1 RandomVariables........................... 56 6.1.1 JointDistributions....................... 58 6.1.2 CharacteristicFunctions.................... 60 6.1.3 Entropy............................ 62 6.2 StochasticProcesses.......................... 64 6.2.1 DistributionEvolutionEquations............... 65 6.2.2 StochasticDifferentialEquations............... 69 6.3 RandomNumberGenerators...................... 70 6.3.1 LinearCongruential...................... 70 6.3.2 LinearFeedback........................ 72 6.4 RandomAlgorithms.......................... 74 6.5 SelectedReferences.......................... 74 6.6 Problems................................ 74 Part Two: Numerical Models 77 7 Finite Differences: Ordinary Differential Equations 81 7.1 NumericalApproximations....................... 81 7.2 Runge KuttaMethods......................... 84 7.3 BeyondRunge Kutta......................... 86 7.4 SelectedReferences.......................... 90 7.5 Problems................................ 90 8 Finite Differences: Partial Differential Equations 92 8.1 HyperbolicEquations:Waves..................... 93 8.2 ParabolicEquations:Diffusion..................... 95 8.3 EllipticEquations:BoundaryValues.................. 98 8.4 SelectedReferences.......................... 105
viii Contents DRAFT 8.5 Problems................................ 105 9 Finite Elements 107 9.1 WeightedResiduals.......................... 107 9.2 Rayleigh RitzVariationalMethods................... 113 9.3 BoundaryConditions.......................... 114 9.4 SolidMechanics............................ 115 9.5 SelectedReferences.......................... 117 9.6 Problems................................ 117 10 Cellular Automata and Lattice Gases 119 10.1 LatticeGasesandFluids........................ 120 10.2 CellularAutomataandComputing................... 124 10.3 SelectedReferences.......................... 126 10.4 Problems................................ 127 11 Computational Geometry 128 11.1 Meshes................................. 128 11.2 Distances................................ 128 11.3 Shapes................................. 129 11.4 Graphs................................. 129 11.5 Folding................................. 130 11.6?.................................... 130 11.7 SelectedReferences.......................... 130 11.8 Problems................................ 130 Part Three: Data-Driven Models 131 12 Function Fitting 135 12.1 ModelEstimation........................... 136 12.2 LeastSquares............................. 137 12.3 LinearLeastSquares.......................... 138 12.3.1 SingularValueDecomposition................. 139 12.4 NonlinearLeastSquares........................ 142 12.4.1 Levenberg MarquardtMethod................ 143 12.5 Errors................................. 144 12.6 Estimation, Fisher Information, and the Cramér Rao Inequality.... 146 12.7 SelectedReferences.......................... 148 12.8 Problems................................ 148 13 Transforms 149 13.1 OrthogonalTransforms........................ 149 13.2 FourierTransforms.......................... 150 13.3 Wavelets................................ 152 13.4 PrincipalComponents......................... 157 13.5 IndependentComponents....................... 158
DRAFT Contents ix 13.6 SelectedReferences.......................... 160 13.7 Problems................................ 160 14 Architectures 161 14.1 Polynomials.............................. 161 14.1.1 PadéApproximants...................... 161 14.1.2 Splines............................. 163 14.2 OrthogonalFunctions......................... 164 14.3 RadialBasisFunctions......................... 168 14.4 Overfitting............................... 169 14.5 CurseofDimensionality........................ 170 14.6 NeuralNetworks............................ 172 14.6.1 BackPropagation........................ 174 14.7 Regularization............................. 175 14.8 SelectedReferences.......................... 177 14.9 Problems................................ 177 15 Optimization 178 15.1 MultidimensionalSearch........................ 179 15.1.1 DownhillSimplex....................... 179 15.1.2 LineMinimization....................... 181 15.1.3 DirectionSet.......................... 182 15.1.4 ConjugateGradient...................... 183 15.1.5 StochasticSearch....................... 185 15.2 LocalMinima............................. 185 15.3 SimulatedAnnealing.......................... 187 15.4 GeneticAlgorithms........................... 188 15.5 TheBlessingofDimensionality.................... 190 15.6 SelectedReferences.......................... 192 15.7 Problems................................ 192 16 Density Estimation 194 16.1 Histogramming,Sorting,andTrees.................. 194 16.2 FittingDensities............................ 197 16.3 Mixture Density Estimation and Expectation-Maximization...... 199 16.4 Cluster-WeightedModeling...................... 203 16.5 ProbabilisticNetworks......................... 210 16.6 SelectedReferences.......................... 210 16.7 Problems................................ 210 17 Constrained Optimization 211 17.1 LagrangeMultipliers.......................... 211 17.2 Optimality............................... 213 17.3 Solvers................................. 213 17.3.1 Penalty............................. 213 17.3.2 AugmentedLagrangian.................... 214
x Contents DRAFT 17.3.3 InteriorPoint......................... 214 17.4 SelectedReferences.......................... 215 17.5 Problems................................ 215 18 Convex Optimization 218 18.1 Kernels................................. 218 18.2 SupportVectorMachines....................... 220 18.2.1 Classification.......................... 220 18.2.2 Regression........................... 224 18.2.3 Clustering........................... 225 18.3 Relaxations............................... 226 18.4 SelectedReferences.......................... 226 18.5 Problems................................ 227 Part Four: Dynamical Models 229 19 Filtering and State Estimation 231 19.1 MatchedFilters............................ 231 19.2 WienerFilters............................. 232 19.3 KalmanFilters............................. 234 19.4 NonlinearityandEntrainment..................... 239 19.5 HiddenMarkovModels........................ 242 19.6 GraphicalNetworks.......................... 247 19.7 SelectedReferences.......................... 247 19.8 Problems................................ 248 20 Linear and Nonlinear Time Series 249 20.1 LinearTimeSeries........................... 250 20.2 TheBreakdownofLinearSystemsTheory.............. 252 20.3 State-SpaceReconstruction...................... 253 20.4 Characterization............................ 258 20.4.1 Dimensions.......................... 259 20.4.2 LyapunovExponents...................... 261 20.4.3 Entropies............................ 262 20.5 Forecasting............................... 265 20.6 SelectedReferences.......................... 269 20.7 Problems................................ 269 21 Control Theory 270 21.1 Problems................................ 276 Appendix 1 Graphical and Mathematical Software 278 A1.1 MathPackages............................. 279 A1.1.1 ProgrammingEnvironments.................. 279 A1.1.2 InteractiveEnvironments................... 281 A1.2 GraphicsTools............................. 285
DRAFT Contents xi A1.2.1 Postscript........................... 285 A1.2.2 XWindows.......................... 290 A1.2.3 OpenGL............................ 294 A1.2.4 Java.............................. 300 A1.2.5 *ification............................ 304 A1.3 Problems................................ 306 Appendix 2 Network Programming 307 A2.1 OSI,TCP/IP,andAllThat...................... 307 A2.2 SocketI/O............................... 308 A2.3 ParallelProgramming......................... 311 Appendix 3 Benchmarking 314 Appendix 4 Problem Solutions 316 A4.1 Introduction.............................. 316 A4.2 LinearAlgebra............................. 316 A4.3 OrdinaryDifferentialandDifferenceEquations............ 316 A4.4 PartialDifferentialEquations...................... 323 A4.5 VariationalPrinciples.......................... 326 A4.6 GeometryandSymmetry........................ 329 A4.7 RandomSystems............................ 329 A4.8 FiniteDifferences:OrdinaryDifferentialEquations.......... 335 A4.9 FiniteDifferences:PartialDifferentialEquations........... 343 A4.10FiniteElements............................ 353 A4.11CellularAutomataandLatticeGases.................. 359 A4.12FunctionFitting............................ 369 A4.13Transforms............................... 374 A4.14Architectures.............................. 380 A4.15Search................................. 387 A4.16DensityEstimation........................... 391 A4.17ConstrainedOptimization....................... 395 A4.18ConvexOptimization.......................... 397 A4.19FilteringandStateEstimation..................... 397 A4.20LinearandNonlinearTimeSeries................... 400 A4.21ControlTheory............................. 404 Bibliography 405 Index 417