Ben Schmidt’s terrific insight into the assumptions that the Fourier transform imposes on sentiment data has been sinking in, and I have a left-field suggestion for anyone who cares to check it out. I plan to investigate it myself when I have the time, but I’ve decided to broadcast it now.

In the imaginary universe of Fourier land, all texts start and end at the same sentiment amplitude. This is clearly incorrect, as I see it.^{1} But what *could* we say about the beginning and end of texts that might hold up?

One possibility is that all texts might start and end with a *flat sentiment curve*. That is, at the *very* beginning and end of a text, we can assume that the valence of words won’t shift dramatically. That’s not clearly incorrect. I think it’s even plausible.

Now consider how we talk about plot most of the time: we speak of rising action (slope positive), falling action (slope negative), and climaxes (local and global maxima). That’s first derivative talk! And the first derivative of a flat curve is always zero. So if the first derivative of a sentiment curve always starts and ends at zero, then at least one objection to the Fourier transform approach can be worked around. For example, we could simply take the first finite difference of a text’s sentiment time series, perform a DFT and low-pass filter, do a reverse transform, and then do a cumulative sum (i.e. a discrete integration) of the result.^{2}

What would that look like?

- Nonetheless, I think there’s some value to remaining agnostic about this for some time still — even now, after the dust has settled a bit. ↩
- You might be able to skip a step or two. ↩

Interesting. I was just mulling a similar idea–take the high-pass filtered recomposition not of the raw sentiments, but of their offset from a linear model, and then recompose with the trend intact. But in either case, I’m not sure I see the advantage–essentially, the Fourier transform is trying to say that to understand the sentiment of the fifth percentile of a book, the 95th percentile conveys the same amount of information as the 15th. I just can’t think of how to make that assumption useful.

But I don’t really believe that things end flatly, in particular. (Beginnings I’m more agnostic about). The moving averages in Syuzhet usually show a slight uptick at the end. Casablanca is I think the perfect example here: most books seem to be finding a way to close on the note of “this is the beginning of a beautiful friendship,” regardless what else just happened. Ulysses sure ends on an upswing according to sentiment scores. OTOH, if endings must be flat, it would definitively prove that Tony Soprano is still alive.

I take your point about upticks at the end, but in the schema I’m imagining,

Ulyssesalmost certainly has a flat ending. It’s literally aclimax, after all. It’s a local maximum, and local maxima are flat.To put this in a slightly different way, the assumption this modified transform makes is that beginnings and endings are always local minima, local maxima, or saddle points. I’m not sure the upticks you’re describing necessarily invalidate that assumption.

But that opens the door for a lot of just-so stories. And you’re certainly right about the “uniform information” assumption, which is very strong. I’m not at all

certainthis is the right approach. I just think it’s one worth examining.OK, fair enough. This seems like it might be a useful assumption for certain models.

I think it might be actually be a pretty powerful way of compensating for the deficiencies of using fitted higher-order polynomials as a smoothing method. (No one has suggested these yet, for good reason, but they might actually work reasonably well in the same way that the Fourier does.)

Yes, I’ll have to look into that as well. But after seeing your graphs in MJ’s most recent post, I feel less urgency about it. There’s just too much interesting local texture that these methods seem to gloss over and that other smoothing methods preserve.