article is missing from feed, but shows up in browse list

plosone.log from clinicaltrials-stage.plos.org for browse.action

plosone.log from clinicaltrials-stage.plos.org for feed

uploaded some logs at jeff's request.

i used the same URLs as listed in the description, but with the clinicaltrials-stage.plos.org host.

the mulgara logs are too long to attach - it probably makes the most sense for you to login to plosstage01.plos.org as topazdev (IM or call me for the password if you need it) and look directly at the mulgara log, using timestamps from the plosone log for reference.

the 'failed to parse date' error is interesting, since the date range is, in fact, being respected despite the error. i believe that at some point eric wrote code to allow the feed to accept various date formats, perhaps this error is a leftover, or the feed looks for a standard date first, throws an error, and then moves on to whatever other date parsing is available?

2007-12-20 13:03:35,235 DEBUG ArticleOtmService(PLoSClinicalTrials)> failed to parse date '2007-12-12' use Date - trying iso8601 format [TP-Processor5 org.plos.article.service.ArticleOtmService]
java.lang.IllegalArgumentException

the XML is the same for all articles:

  <pub-date pub-type="epub">
    <day>12</day>
    <month>12</month>
    <year>2007</year>
  </pub-date>

the date parsing does fall back so the Exception isn't relevant.

Browse & Feeds query the database differently:

• Browse gets a list of all Article meta data and builds a map
• Feed does an explict OTM query

looking at the Mulgara logs & indexes is probably fruitful.

for ntds 12/26 pub date:

The following articles were included: 32 (frontmatter) and 88, 103, 111 (research)

The following articles were missing: 149, 159, 161, and 168 (all frontmattter)

i'm wondering if there's something specific that the feed is looking for that is missing in the XML (and thus mulgara) for the articles that don't show up in the feed?

this is not the same as the bug in #762

Related to #745 ehcache cleanup.

Is this still a problem?

yes. it's still an issue in 0.9.

example. the following articles were published, appear in browse, but do not appear in the feed:

• pntd.0000270
• pntd.0000268
• pntd.0000267

http://www.plosntds.org/article/browse.action?field=date http://feeds.feedburner.com/plosntds/NewArticles

anecdotally, this almost always, or perhaps always, happens with front matter (non-research) articles.

Hang on. These are not research articles, but editorials?

yes. the feed should include all articles published. currently it includes *most* articles published.

sometimes research articles are missed as well, but it's more rare.

i expect there's something in xml that almost all research articles have, but many front matter articles don't, which is causing them to elude the feed.

Jon, please take a look at this too.

I would test the 'internal' URLs before testing it via feed burner.

feedburner is not the issue.

http://plosntds-stage.plos.org/article/browse.action?field=date http://plosntds-stage.plos.org/article/feed?endDate=2008-08-01

if you prefer :)

Moving to next milestone.

I tested with these three articles and they are showing in feed now.

• pntd.0000270
• pntd.0000268
• pntd.0000267

Probably fixed with #745

Please confirm.

it's hard to confirm since it's an intermittent problem, and i've never understood why this was happening.

can you explain what you think the bug was and how #745 fixed it?

perhaps that will help me come up with a test case.

if not, we can wait until after deployment to see if this is still a problem.

#745 fixed feed cache invalidation. My impression was that this problem might have been caused by feeds not being properly invalidated on new article publish or virtual journal changes.

I agree that we should wait until deployment to see if this problem is still present.

there is no feed invalidation in 0.9 - there's just a longish time out.

however, iirc this problem was reproduceable even after emptying the feed cache, or when creating a feed with parameters that did not match any keys in the cache...

i'll confirm with susanne...

apparently this has been fixed in 0.9 - james and susanne report no recent occurrence of this bug, and the feed for dates which definitely had this issue in the past is now showing all article correctly.

Milestone 0.9.1 deleted