Update PortIndex incrementally so it is always current

[frstan@…]

On Nov 9, 2008, at 9:03 AM, William Davis wrote: Commit by ryandesign@… :: r41701 /trunk/base/src/port/portindex.tcl: (link) ​http://trac.macports.org/changeset/41701 portindex.tcl: Generate the port index in a temporary file first, then replace the PortIndex file all at once; fixes #16234

If this is the case cant we easily update the index more than twice a day now? Perhaps even every half hour?

portindex is an annoyingly expensive operation. I don't know if the added load on the macports servers would be worth it?

If someone gets around to implementing an incremental portindex option (say we store the index in an sqlite database and we add a post-commit hook that just updates the index entries for the ports that were just committed) then we could probably have an always up-to-date index without the huge resource requirements.

-- Daniel J. Luke ticket filed by W Davis

[wsiegrist@…]

As inefficient as it is, it does not make a noticeable impact on server load. I can simply increase the frequency if thats what is desired. I'll have to see how long it takes these days to see if half-hourly is possible, but I believe hourly is definitely possible. This ticket seems to be for making indexing incremental, so I'm removing it from server/hosting and marking it as a base enhancement. If portmgr wants more frequent indexes, the can email or file a specific ticket.

[blb@…]

If #16235 gets implemented, then portindex could run for each commit. However, if that is done, then we see a commit to PortIndex every time a port is updated, doubling the number of commits to the repo.

Would it be lots of work to simply pull PortIndex out of subversion, or would it be simpler to finish the SQL index and have that sync to the MacPorts server in a port sync?

[raimue (Rainer Müller)]

If portindex gets faster, we could even generate it on the client side only by adding it to 'port sync'.

Isn't #16235 basically the same request as this ticket?

[blb@…]

Replying to raimue@…:

If portindex gets faster, we could even generate it on the client side only by adding it to 'port sync'.

Wouldn't that require us to either pay attention to what rsync is updating, or to crawl through afterwards looking for newer files? The latter works also for ​file:// synced via svn.

[raimue (Rainer Müller)]

Replying to blb@…:

Replying to raimue@…:

If portindex gets faster, we could even generate it on the client side only by adding it to 'port sync'.

Wouldn't that require us to either pay attention to what rsync is updating, or to crawl through afterwards looking for newer files? The latter works also for ​file:// synced via svn.

Yes, my idea was to store the last modified timestamp of a Portfile in the PortIndex and only update the entry if the file is newer, by using the incremental portindex (as described in #16235). This should work for every sync method.

[jmroot (Joshua Root)]

Updating indices on sync as of r66846.