Showerthought: I fed my model the answer key for 6 weeks before I got why it failed on live data
I was building a small classifier for a client's support tickets and I kept patting myself on the back because my accuracy hit 97% in testing. Then last Tuesday the thing fell apart on real tickets and it was flagging refund requests as praise. Turns out I had been splitting my data by random rows, so the same ticket threads showed up in both my train and test sets and the model was just memorizing near copies. A buddy at a meetup in Columbus looked at my script for 30 seconds and said "you leaked your own test set" and I wanted to crawl under the table. I redid the split by date instead of by row and my honest number dropped to 81%, which was a punch to the gut but at least it was real. Now I get why every tutorial screams about leakage, it just never clicked until it bit me. How do you all handle time based splits when your data comes in uneven batches?
When you say the data comes in uneven batches, do you mean some weeks have a ton of tickets and others barely any, or is it more that the topics themselves cluster in bursts? The reason I ask is those two problems need pretty different fixes. If it's just volume, a simple date cutoff usually holds up fine. But if refund waves or outages make whole weeks look nothing like the rest, then a single split won't tell you much about how the model does when the next weird week hits. Have you tried rolling your split forward in chunks (train on months 1-3, test on 4, then shift) to see if that 81% stays steady or swings around? That number moving a lot between chunks would tell you more than one split ever could.