***** TORNADO BACK END ***** Copyright (c) 1999-2008 Highwinds Software LLC All Rights Reserved. www.highwinds-software.com software@highwinds-software.com 1.866.NNTP.119 ------------------------------------------------------------------------------- Tornado Backend 1.1.1 New configuration file settings: HeaderFeedQueueCacheSize <#> where <#> is the number of article header references to cache in RAM. Each reference is 28 bytes. Default is 16K references. Max is 16M references. HeaderFeedDiskCacheSize <#> where <#> is the number of article header references cached in RAM before an actual write to disk occurs. Default is 16K references. Max is 256K ------------------------------------------------------------------------------- Tornado Backend 1.1.0 New features & files ----- A new queuing mechanism has been built for the BE to feed article headers to each FE. This mechanism fixes problems where FEs could fail to receive articles or FEs would erroneously receive duplicates of very old articles. The new mechanism creates a new queue file named tornadobe.hfd. It also creates one new file per FE named with the feed name and with the extension .hfp, which stands for Header Feed Position. To convert from your existing queue data files, an upgrade procedure is defined below. The size of the HFD file is determined using the existing value for the size of the dummy feed files. You may have specified the size of the dummy feed with DummyFrontEndQueueSize or with MasterFrontEndQueueSize. Whichever is set is the one used when the HFD file is created. The value is in KB, with a maximum size of 4000000000 KB (4TB, 125 billion article references). However, do not change this value until after the upgrade procedure has been completed. When the upgrade procedure is executed, the HFD will be created using the existing path for the dummy queues and using the existing queue size. Anything in the dummy queues will be copied to the new HFD queue. Be sure to check available disk space when running the upgrade tool. If necessary, the df.1/df.2 files can be copied to a different filesystem and then sym-linked to from their original locations. The HFD can be expanded at any time simply by increasing the size specified in the conf file. When restarted, TornadoBE/Stormcellar will expand the HFD file to the given size and the additional space will start to be used when the previously existing bytes in the HFD file are filled. NOTE: The HFD file is not a sparse file. On startup, the full size specified in the config is taken. New hfp files will be created in the directory specified in each Feed's TornadoQueuePath setting in the feeds.conf file. When a new FE feed is configured, on startup the TBE will place that feed as far back as possible in the queue so that the new FE will receive the maximum history available from the TBE. When a feed is upgraded using the procedure defined below, the feed is placed at the head of the queue as if it already has everything the TBE has to offer. Spool files, active files, MID files, H files, track files and Overview files are unaffected by this upgrade. Also, no FE changes are required. Upgrade procedure ----- Note: The upgrade tool uses the existing TBE config files, which may contain paths relative to the run location of the TBE, which is generally in the tornado install's bin/ dir. The upgrade tool must be run from the same directory that the TBE runs. STOP the TBE No configuration changes should be made Make sure you have enough space on your filesystem(s) for the new data files. The new HFD data file will be the size specified for the current df.1/df.2 data files, so the single HFD data file will be twice the size of one of the df data files. Execute the upgradeToHFD utility from the bin directory where the TBE executable is located If the upgradeToHFD utility reports that it worked, move your current TBE binary out of the way and put the new TBE binary in place. Start the new TBE All FE queues are positioned at the front of the HFD queue. If you need to roll any back, this can be done manually. See below for details. Once upgrading is complete and you are satisified with the performance, df.1 & df.2 can be deleted Downgrade procedure ----- Note: The downgrade tool uses the existing TBE config files, which may contain paths relative to the run location of the TBE, which is generally in the tornado install's bin/ dir. The downgrade tool must be run from the same directory that the TBE runs. STOP the TBE If you increased the size of the HFD beyond 4GB, you will need to change your config file so that DummyFrontEndQueueSize/MasterFrontEndQueueSize is less than 4GB. Make sure you have enough space on your filesystem(s) for the df.1/df.2 files to be created. Execute the downgradeFromHFD utility from the bin directory where the TBE executable is located If the downgradeFromHFD utility reports that it worked, move your new TBE binary out of the way and restore the previous TBE binary. Start the TBE Note that no more than 70M article references can be copied from the HFD into the dummy queues as the dummy queues have a maximum of 35M articles each. All FE queues are positioned at the front of the df.1 queue. If you need to roll any back, this can be done manually. See below for details. Once downgrading is complete and you are satisified with the performance, the .hfd file and the .hfp files can be deleted How to manually adjust FE depth ---- The file header_queue.control can be updated at run time with instructions on how to alter any FE's queue position. Format of file: # comment incomingFeedName can be a number up to 10 digits in length or may say "max" Examples: mrhowell 50 skipper max # Here is a comment The file should be created in the same directory as the active files. Once you have header_queue.control as you like it, execute the bundled script "reload" to cause the BE to read the file and process the requests. The log will show progress with LOG_INFO messages and errors with LOG_ERR messages. Once the commands in header_queue.control are executed, header_queue.control will be deleted. ------------------------------------------------------------------------------- Tornado Backend 1.0.5.448 - Fixed memory leak when article headers are queued - Added support for friendly Diablo retention ------------------------------------------------------------------------------- Tornado Backend 1.0.5.424 - Added commamnd line argument to tornado to suppress writing of $/log/stats.weights ------------------------------------------------------------------------------- Tornado Backend 1.0.5.395 - Added XBEREPLACE - allows for the reconstruction of a spool by number. ------------------------------------------------------------------------------- Tornado Backend 1.0.5.394 - Added deeper retention to history. ------------------------------------------------------------------------------- Tornado Backend 1.0.5.390 - Added Terabyte directive for spool sizes. ------------------------------------------------------------------------------- Tornado Backend 1.0.4.380 - Semantical fix with xover, xover # was interpreted as #- that has been corrected - Eliminate/reduce article loss during server shutdown. ------------------------------------------------------------------------------- Tornado Backend 1.0.4.371 - XOVERs are now correctly bandwidth-limited for MaxBytesPerSecond and FeedMaxBytesPerSecond. - The -strict option will prevent users who send bad commands from being disconnected. - More robust handling of huge overview lines. - Bug fix: Spools larger than 32Gb are now handled correctly when starting with -nocreate. - Added -veritas flag which enables VXFS Direct I/O if -direct is also set. ------------------------------------------------------------------------------- Tornado Backend 1.0.4.370 - Fixed PID file misspelling. - Dummy feed size increase to 2Gb by default. - New slave queueing architecture saves disk space for front-end queues. - "Terabytes" now recognized in spool objects. -------------------------------------------------------------------------------- 1.0.4 - Added a new directive called XrefMaster. Use of this directive allows only "trusted" host names that perform article number generation to store articles. The directive XrefMaster takes one or more wildcard arguments. At least one of these arguments have to match the host field in the Xref: header field when the server is set to XrefAction Pass mode. A match failure in parse mode forces an article rejection with code 0x. This directive would apply to cyclone.conf, typhoon.conf, twister.con, and tornado_be.conf. - Fix a random memory corruption problem which has the potential to cause core dumps. - Master/Slave Frontend queues added. Under this new paradigm DummyQueuePath and DummyQueueSize have been depricated and replaced with MasterQueuePath and MasterQueueSize. This queue is used as a repository for all front ends. Each additional frontend simply maintains a master block of 16 bytes, instead of copying the entire queue. This system is not backwards compatible, and will update automatically. When the (old) queue empties, it will be replaced with a .slave file and unlinked. This will happen if/when it empties. To force this, momentarily turn off your cyclone, which will allow all queues to flush. NOTE: ONCE THIS HAPPENS THE SERVER CANNOT BE ROLLED BACK TO ANY PREVIOUS VERSION - SendToTornados has been depricated, now the -backfill command-line option is used. This will scour the spools and place all articles into the MasterQueue (defined in tornado_be.conf) for use by all frontends. -------------------------------------------------------------------------------- 1.0.3 - Added Front End stats collection. Stats are shown in stats.frontend file. See the file README.stats for the stats description. - Overview pre-caching added, this allows greater scaling in the number of Front Ends a Back End can support (effectively unlimited for shared spools). The cache is sized with the configuration option: OverviewPreCache # where # is size in Kilobytes (0-100,000) (a size of zero disables it) - Dramatic reduction in RAM footprint - AdaptiveSubscriptionPolicy [none|bool|weighted] specified in config file turns on that type of adaptive storage for all spools - -nonblockingclose option to attempt to recover thread resources when closing reader connections. This is useful when a server is receiving many multiple connect/disconencts. - Support for raw devices. use "AccessMode RAW" in a spool object and specify the raw device as the Path. raw devices must be formatted/labeled with the supplied tools/rawtool - Cascade feeding supported. Defining a cascade feed will cause articles that are just about to expire to be send to a downstream server (stormcellar) for archival. Feed Object format: OutgoingHostName outgoing_hostname OutgoingFeedName cascade_name Port 119 AdaptiveCascade true/false - Multiple cascade feeding supported, new directives in .conf are: NumberOfCascadeStreams # CascadeDepthPercent # - Large spools now supported, Spool sizes may now be expressed in "Megabytes" and "Gigabytes", and be up to 2 Terabytes in size. - Weights are exposed in stats.weights - Stormcellar Cascade server feed implemented, stats are shown in stats.cascade - All XOver lines are now individually validated for correctness - -keyinfo option in validate -------------------------------------------------------------------------------- 1.0.2 - Connect limit demoted from Error to Info (-detailed to see it) - LogOORArticles [true/false] and DiscardOORArticles [true/false] added. This will affect articles in groups of more than 1000 articles this catches mis-numbered articles. - Crashing when syslog > 2G fixed (Linux only) - Empty syslogs no longer emit critical errors - Enhanced logging for syslog messages that displayed (m) for non-Solaris Operating Systems -------------------------------------------------------------------------------- 1.0.1 - -crossplatformspools allows shared-spool support. WARNING- turning this on WILL DESTROY backward compatibility on non-network order machines (linux/FreeBSD) - CIDR code optimized - ChainMessageIDRequests True|False (default true) wether or not message ID requests will be passed back to a chaining host - Chaining timeouts loosened up for loaded backends. - -nohistory flag added to trust all backend content - -strict flag for strict/narrow NNTP interpretation - cat.cat file corrupt bug fixed - large spool supported -------------------------------------------------------------------------------- 1.0.0 - Added DummyFrontendQueuePath/DummyFrontendQueueSize so back-end can establish a dummy-feed to copy into new front-ends - Added TornadoAccessMode to config. can not explicitly set how front-ends will access the shared spools - Fixed upstream cancel propogation - Propogate moderated active file list - Added SharedSpools to feed - Added TornadoPath to config -------------------------------------------------------------------------------- Copyright (c) 1999-2008 Highwinds Software LLC All Rights Reserved. www.highwinds-software.com software@highwinds-software.com