Sign Up
Log In
Log In
or
Sign Up
Places
All Projects
Status Monitor
Collapse sidebar
openSUSE:Step:15
rsyslog.17135
0001-imfile-large-refactoring-of-complete-modul...
Overview
Repositories
Revisions
Requests
Users
Attributes
Meta
File 0001-imfile-large-refactoring-of-complete-module.patch of Package rsyslog.17135
From 6c0a5722641ce0bad583279986edf8f7bc768354 Mon Sep 17 00:00:00 2001 From: Rainer Gerhards <rgerhards@adiscon.com> Date: Tue, 6 Mar 2018 15:49:27 +0100 Subject: [PATCH 1/7] imfile: large refactoring of complete module This commit greatly refactors imfile internal workings. It changes the handling of inotify, FEN, and polling modes. Mostly unchanged is the processing of the way a file is read and state files are kept. This is about a 50% rewrite of the module. Polling, inotify, and FEN modes now use greatly unified code. Some differences still exists and may be changed with further commits. The internal handling of wildcards and file detection has been completely re-written from scratch. For example, previously when multi-level wildcards were used these were not reliably detected. The code also now provides much of the same functionality in all modes, most importantly wildcards are now also supported in polling mode. The refactoring sets ground for further enhancements and smaller refactorings. This commit provides the same feature set that imfile had previously and all existing CI tests pass, as do some newly created tests. Some specific changes: - bugfix: module parameter "sortfiles" ignored This parameter only works in Solaris FEN mode, but is otherwise ignored. Most importantly it is ignored under Linux. fixes https://github.com/rsyslog/rsyslog/issues/2528 - bugfix: imfile did not pick up all files when not present at startup fixes https://github.com/rsyslog/rsyslog/issues/2241 fixes https://github.com/rsyslog/rsyslog/issues/2230 fixes https://github.com/rsyslog/rsyslog/issues/2354 - bugfix: directories only support "*" wildcard, no others fixes https://github.com/rsyslog/rsyslog/issues/2303 - bugfix: parameter "sortfiles" did only work in FEN mode fixes https://github.com/rsyslog/rsyslog/issues/2528 - provides the ability to dynamically add and remove files via multi-level wildcards see also https://github.com/rsyslog/rsyslog/issues/1280 - the state file name currently has been changed to inode number This will further be worked on in upcoming PRs see also https://github.com/rsyslog/rsyslog/issues/2231 - some enhancements were also done to CI tests, most importantly they were made more compatibile with BSD Note that most of the mentioned bug fixes cannot be applied to older versions, as they fix design issues which are solved by the refactoring. Thus there are not separate commits for them. Distro maintainers: you need to decide to apply this patch as whole or not. Believe me, it is not worth the effort to try to extract specific patches from this commit. There is a good reason we do not have multiple commits. closes https://github.com/rsyslog/rsyslog/issues/2359 (cherry picked from commit ab1bd8c01b94d184125cbccb897b9319253a0a5b) --- IMFILE | 131 + plugins/imfile/imfile.c | 3194 +++++++-------------- tests/Makefile.am | 22 +- tests/diag.sh | 39 +- tests/imfile-discard-truncated-line.sh | 2 +- tests/imfile-endregex-save-lf-persist.sh | 2 +- tests/imfile-endregex-save-lf.sh | 2 +- tests/imfile-endregex-timeout-none.sh | 4 +- tests/imfile-endregex-timeout-with-shutdown.sh | 2 +- tests/imfile-endregex-timeout.sh | 2 +- tests/imfile-file-not-found-error.sh | 4 +- tests/imfile-readmode2-with-persists.sh | 4 + tests/imfile-readmode2.sh | 37 +- tests/imfile-rename-while-stopped.sh | 33 + tests/imfile-rename.sh | 12 +- tests/imfile-truncate-line.sh | 2 +- tests/imfile-wildcards-dirs-multi.sh | 2 +- tests/imfile-wildcards-dirs-multi2.sh | 3 +- tests/imfile-wildcards-dirs-multi3.sh | 5 +- tests/imfile-wildcards-dirs-multi4.sh | 4 +- tests/imfile-wildcards-dirs-multi5-polling.sh | 77 + tests/imfile-wildcards-dirs-multi5.sh | 82 + tests/imfile-wildcards-dirs.sh | 2 +- tests/imfile-wildcards.sh | 32 +- tests/testsuites/imfile-wildcards-dirs-multi.conf | 2 + tests/testsuites/imfile-wildcards.conf | 39 - 26 files changed, 1517 insertions(+), 2223 deletions(-) create mode 100644 IMFILE create mode 100755 tests/imfile-rename-while-stopped.sh create mode 100755 tests/imfile-wildcards-dirs-multi5-polling.sh create mode 100755 tests/imfile-wildcards-dirs-multi5.sh delete mode 100644 tests/testsuites/imfile-wildcards.conf diff --git a/IMFILE b/IMFILE new file mode 100644 index 000000000..e960dd35b --- /dev/null +++ b/IMFILE @@ -0,0 +1,131 @@ +Use Cases +--------- + +Monitor: +[M1] /dir/*/*.log ; tag="monitor1" +[M2] /dir/abc/a.* ; tag="monitor2" +[M3] /dir/*/a/*.txt ; tag="monitor3" +[M4] /di?/*/b/*.txt ; tag="monitor4" + +- if a.log is created, we have kind of a conflict: create two monitors then? + (sounds best so far) +- /dir/abc may not initially exist, but later on created + -> same problem for any path depth (polling? multiple inotify? fail?) + +Let's assume /dir/abc exist, but nothing else under /dir + +Setup initial inotify: +/dir [M1] +/dir [M3] +/dir/abc [M1] +/dir/abc [M2] +/dir/abc [M3] +/dir/abc [M4] +Rule? The parent of each component must be monitored +/ [M4] -- really??? Means we need to monitor whole FS tree! +/dir [M2] - /dir/abc may not yet exists but may be created later on + +Q: can we combine /dir/abc? +A: well, we need to know what shall be monitored. e.g. + +/dir/abc/a.logfile is created +watch /dir/abc must be evaluated +does not match [M1], but [M2]! +Event for /dir/abc is generated + need to check [M1], [M2] + +/dir/def is created +watch /dir must be evaluated --> [M1], [M2], [M3], [M4] must be checked +matches [M1],[M3],[M4] + create new watch + /dir/def [M1, M3, M4] - this watch should not exist if all is right [-> assert()] + +/dir/def/new.log is created +watch /dir/def must be evaluated --> [M1, M3, M4] + create new file watch + /dir/def/new.log [M1] + +/dir/abc/a.log is created +watch /dir/abc must be evaluated --> [M1, M2, M3, M4] + create new file watch + /dir/abc/a.log [M1, M2] + +At each level, I must know which child components to watch out for, and which +monitor/listener they belong to. Comparison must be done on immediate-child level. + +Data Structures +-------------- + +Monitors +........ +- path needs to be split into individial components --> array + +File Watches +............ +- contains active file state +- one file watch per file per monitor (/dir/abc/a.log sample with two active file entries) +- multiple entries are not properly supported by inotify (and possibly FEN) + --> need to multiplex ourselfes based on single notifcation + +Dir Watches +........... +- contains active directory state +- used to discover new to-be-monitored files + +Problems +-------- +- we need to find, for each level, the proper "next component" that we need to check + against (fnmatch on component?) +- this must be done om a per-monitor basis +- one entry per monitored file, but a list of monitors (and component names to check against) + --> how do we efficiently describe the monitor so that we can do an O(1) name comparison? + --> we can NOT assume that the full path matches, just component-by-component! + --> we may NOT get notifications if path level are rapidly created, as we may not + have setup the watch sufficiently fast - so we need to poll the full tree + in any case + +- state files: different state if the same file is monitored by more than one monitor + --> or should we flag this as an error case? + +Poll vs. Watch +-------------- +Polling is always needed (due to watch racieness). So it probably is best to have some generic +poll code, and use watches just to reduce the amount of polling. That also means that +poll mode can do exactly the same the notification mode can do! + + +Procedures +---------- + +e.g. /dir/aaa is created +this will be reported on /dir watch, which is watched by all 4 monitors +foreach monitor: + check if next-level component matches - this is the case for [M1, M2, M3] + if so: + create new/update existing entry (existing for all but first!) + arm watch for this entry (if newly created) + do a full poll of that entry and its subentries + AS FAR AS THE MONITOR IN QUESTION is affected: + M1 will scan subdir for "*.log" + M2 will scan for "abc", if it exists scan for "a.*" + M3 will scan for "a", if it exists scan for "*.txt" + --> rest of the path components need to be scanned, files + processed for each monitor + this is a recursive function + +The above-mentioned poll process is fundamental, and can occur on each level +of the file system/monitor definitions. It needs to work both on active monitors as well +as the configured path components. Does ist make sense to have them in a single structure? +Is a tree more useful than an dirs table? + +now the opposite: +e.g. /dir/aaa is deleted +this will be reported on /dir watch, which is watched by all 4 monitors +check if "aaa" exists as active(dynamic) entry +if so: + check for sub-entries, initiate delete events if any exists + (TODO: is MOVE here a problem?) + delete "aaa" entry + Note: we MUST NOT remove entries that are statically configured + (e.g. /dir in our samples, even if it is deleted - it may re-appear) + diff --git a/plugins/imfile/imfile.c b/plugins/imfile/imfile.c index 950b62918..a5d20823f 100644 --- a/plugins/imfile/imfile.c +++ b/plugins/imfile/imfile.c @@ -5,7 +5,7 @@ * * Work originally begun on 2008-02-01 by Rainer Gerhards * - * Copyright 2008-2017 Adiscon GmbH. + * Copyright 2008-2018 Adiscon GmbH. * * This file is part of rsyslog. * @@ -61,8 +61,9 @@ #include "stringbuf.h" #include "ruleset.h" #include "ratelimit.h" +#include "parserif.h" -#include <regex.h> // TODO: fix via own module +#include <regex.h> MODULE_TYPE_INPUT /* must be present for input modules, do not remove */ MODULE_TYPE_NOKEEP @@ -81,11 +82,7 @@ static int bLegacyCnfModGlobalsPermitted;/* are legacy module-global config para #define NUM_MULTISUB 1024 /* default max number of submits */ #define DFLT_PollInterval 10 - -#define INIT_FILE_TAB_SIZE 4 /* default file table size - is extended as needed, use 2^x value */ -#define INIT_FILE_IN_DIR_TAB_SIZE 1 /* initial size for "associated files tab" in directory table */ #define INIT_WDMAP_TAB_SIZE 1 /* default wdMap table size - is extended as needed, use 2^x value */ - #define ADD_METADATA_UNSPECIFIED -1 /* @@ -102,54 +99,6 @@ static int bLegacyCnfModGlobalsPermitted;/* are legacy module-global config para #define GLOB_BRACE 0 #endif -/* this structure is used in pure polling mode as well one of the support - * structures for inotify. - */ -typedef struct lstn_s { - struct lstn_s *next, *prev; - struct lstn_s *masterLstn;/* if dynamic file (via wildcard), this points to the configured - * master entry. For master entries, it is always NULL. Only - * dynamic files can be deleted from the "files" list. */ - uchar *pszFileName; - uchar *pszDirName; - uchar *pszBaseName; - uchar *pszTag; - size_t lenTag; - uchar *pszStateFile; /* file in which state between runs is to be stored (dynamic if NULL) */ - int readTimeout; - int iFacility; - int iSeverity; - int maxLinesAtOnce; - uint32_t trimLineOverBytes; - int nRecords; /**< How many records did we process before persisting the stream? */ - int iPersistStateInterval; /**< how often should state be persisted? (0=on close only) */ - strm_t *pStrm; /* its stream (NULL if not assigned) */ - sbool bRMStateOnDel; - sbool hasWildcard; - uint8_t readMode; /* which mode to use in ReadMultiLine call? */ - uchar *startRegex; /* regex that signifies end of message (NULL if unset) */ - sbool discardTruncatedMsg; - sbool msgDiscardingError; - regex_t end_preg; /* compiled version of startRegex */ - uchar *prevLineSegment; /* previous line segment (in regex mode) */ - sbool escapeLF; /* escape LF inside the MSG content? */ - sbool reopenOnTruncate; - sbool addMetadata; - sbool addCeeTag; - sbool freshStartTail; /* read from tail of file on fresh start? */ - sbool fileNotFoundError; - ruleset_t *pRuleset; /* ruleset to bind listener to (use system default if unspecified) */ - ratelimit_t *ratelimiter; - multi_submit_t multiSub; -#ifdef HAVE_INOTIFY_INIT - uchar* movedfrom_statefile; - __u32 movedfrom_cookie; -#endif -#if defined(OS_SOLARIS) && defined (HAVE_PORT_SOURCE_FILE) - struct fileinfo *pfinf; - sbool bPortAssociated; -#endif -} lstn_t; static struct configSettings_s { uchar *pszFileName; @@ -170,6 +119,7 @@ struct instanceConf_s { uchar *pszDirName; uchar *pszFileBaseName; uchar *pszTag; + size_t lenTag; uchar *pszStateFile; uchar *pszBindRuleset; int nMultiSub; @@ -180,6 +130,7 @@ struct instanceConf_s { sbool bRMStateOnDel; uint8_t readMode; uchar *startRegex; + regex_t end_preg; /* compiled version of startRegex */ sbool discardTruncatedMsg; sbool msgDiscardingError; sbool escapeLF; @@ -195,9 +146,56 @@ struct instanceConf_s { }; +/* file system objects */ +typedef struct fs_edge_s fs_edge_t; +typedef struct fs_node_s fs_node_t; +typedef struct act_obj_s act_obj_t; +struct act_obj_s { + act_obj_t *prev; + act_obj_t *next; + fs_edge_t *edge; /* edge which this object belongs to */ + char *name; /* full path name of active object */ + char *basename; /* only basename */ //TODO: remove when refactoring rename support + //char *statefile; /* base name of state file (for move operations) */ + int wd; +#if defined(OS_SOLARIS) && defined (HAVE_PORT_SOURCE_FILE) + struct fileinfo *pfinf; + sbool bPortAssociated; + int is_deleted; /* debugging: entry deleted? */ +#endif + time_t timeoutBase; /* what time to calculate the timeout against? */ + /* file dynamic data */ + int in_move; /* workaround for inotify move: if set, state file must not be deleted */ + ino_t ino; /* current inode nbr */ + strm_t *pStrm; /* its stream (NULL if not assigned) */ + int nRecords; /**< How many records did we process before persisting the stream? */ + ratelimit_t *ratelimiter; + multi_submit_t multiSub; +}; +struct fs_edge_s { + fs_node_t *parent; + fs_node_t *node; /* node this edge points to */ + fs_edge_t *next; + uchar *name; + uchar *path; + act_obj_t *active; + int is_file; + int ninst; /* nbr of instances in instarr */ + instanceConf_t **instarr; +}; +struct fs_node_s { + fs_edge_t *edges; +}; + + /* forward definitions */ -static rsRetVal persistStrmState(lstn_t *pInfo); +static rsRetVal persistStrmState(act_obj_t *); static rsRetVal resetConfigVariables(uchar __attribute__((unused)) *pp, void __attribute__((unused)) *pVal); +static rsRetVal ATTR_NONNULL(1) pollFile(act_obj_t *act); +static int ATTR_NONNULL() getBasename(uchar *const __restrict__ basen, uchar *const __restrict__ path); +static void ATTR_NONNULL() act_obj_unlink(act_obj_t *const act); +static uchar * ATTR_NONNULL(1, 2) getStateFileName(const act_obj_t *, uchar *, const size_t); +static int ATTR_NONNULL() getFullStateFileName(const uchar *const, uchar *const pszout, const size_t ilenout); #define OPMODE_POLLING 0 @@ -211,70 +209,23 @@ struct modConfData_s { int readTimeout; int timeoutGranularity; /* value in ms */ instanceConf_t *root, *tail; - lstn_t *pRootLstn; - lstn_t *pTailLstn; + fs_node_t *conf_tree; uint8_t opMode; sbool configSetViaV2Method; sbool sortFiles; + sbool normalizePath; /* normalize file system pathes (all start with root dir) */ sbool haveReadTimeouts; /* use special processing if read timeouts exist */ + sbool bHadFileData; /* actually a global variable: + 1 - last call to pollFile() had data + 0 - last call to pollFile() had NO data + Must be manually reset to 0 if desired. Helper for + polling mode. + */ }; static modConfData_t *loadModConf = NULL;/* modConf ptr to use for the current load process */ static modConfData_t *runModConf = NULL;/* modConf ptr to use for the current load process */ -/* Dynamic File support for inotify / fen mode */ -/* we need to track directories */ -struct dirInfoFiles_s { /* associated files */ - lstn_t *pLstn; - int refcnt; /* due to inotify's async nature, we may have multiple - * references to a single file inside our cache - e.g. when - * inodes are removed, and the file name is re-created BUT another - * process (like rsyslogd ;)) holds open the old inode. - */ -}; -typedef struct dirInfoFiles_s dirInfoFiles_t; - -/* This structure is a dynamic table to track file entries */ -struct fileTable_s { - dirInfoFiles_t *listeners; - int currMax; - int allocMax; -}; -typedef struct fileTable_s fileTable_t; - -/* The dirs table (defined below) contains one entry for each directory that - * is to be monitored. For each directory, it contains array which point to - * the associated *active* files as well as *configured* files. Note that - * the configured files may currently not exist, but will be processed - * when they are created. - */ -struct dirInfo_s { - uchar *dirName; - uchar *dirNameBfWildCard; - sbool hasWildcard; -/* TODO: check if following property are inotify only?*/ - int bDirType; /* Configured or dynamic */ - fileTable_t active; /* associated active files */ - fileTable_t configured; /* associated configured files */ - int rdiridx; /* Root diridx helper property */ -#if defined(OS_SOLARIS) && defined (HAVE_PORT_SOURCE_FILE) - struct fileinfo *pfinf; - sbool bPortAssociated; -#endif -}; - -#if defined(HAVE_INOTIFY_INIT) || (defined(OS_SOLARIS) && defined (HAVE_PORT_SOURCE_FILE)) -/* needed for inotify / fen mode */ -typedef struct dirInfo_s dirInfo_t; -static dirInfo_t *dirs = NULL; -static int allocMaxDirs; -static int currMaxDirs; -#endif /* #if defined(HAVE_INOTIFY_INIT) || (defined(OS_SOLARIS) && defined (HAVE_PORT_SOURCE_FILE)) --- */ - -/* the following two macros are used to select the correct file table */ -#define ACTIVE_FILE 1 -#define CONFIGURED_FILE 0 - #ifdef HAVE_INOTIFY_INIT /* We need to map watch descriptors to our actual objects. Unfortunately, the * inotify API does not provide us with any cookie, so a simple O(1) algorithm @@ -285,9 +236,7 @@ static int currMaxDirs; */ struct wd_map_s { int wd; /* ascending sort key */ - lstn_t *pLstn; /* NULL, if this is a dir entry, otherwise pointer into listener(file) table */ - int dirIdx; /* index into dirs table, undefined if pLstn == NULL */ - time_t timeoutBase; /* what time to calculate the timeout against? */ + act_obj_t *act; /* point to related active object */ }; typedef struct wd_map_s wd_map_t; static wd_map_t *wdmap = NULL; @@ -304,10 +253,6 @@ struct fileinfo { }; static int glport; /* Static port handle for FEN api*/ - -/* Need these function to be declared on top */ -static rsRetVal fen_DirSearchFiles(lstn_t *pLstn, int dirIdx); -static rsRetVal fen_processEventDir(struct file_obj* fobjp, int dirIdx, int revents); #endif /* #if OS_SOLARIS -------------------------------------------------- */ static prop_t *pInputName = NULL; @@ -319,6 +264,7 @@ static struct cnfparamdescr modpdescr[] = { { "readtimeout", eCmdHdlrPositiveInt, 0 }, { "timeoutgranularity", eCmdHdlrPositiveInt, 0 }, { "sortfiles", eCmdHdlrBinary, 0 }, + { "normalizepath", eCmdHdlrBinary, 0 }, { "mode", eCmdHdlrGetWord, 0 } }; static struct cnfparamblk modpblk = @@ -362,20 +308,32 @@ static struct cnfparamblk inppblk = #include "im-helper.h" /* must be included AFTER the type definitions! */ -#ifdef HAVE_INOTIFY_INIT -/* support for inotify mode */ +#if 0 // Code we can potentially use for new functionality // TODO: use or remove +//TODO add a kind of portable asprintf: +static const char * ATTR_NONNULL() +gen_full_name(const char *const dirname, const char *const name) +{ + const size_t len_full_name = strlen(dirname) + 1 + strlen(name) + 1; + char *const full_name = malloc(len_full_name); + if(full_name == NULL) + return NULL; + snprintf(full_name, len_full_name, "%s/%s", dirname, name); + return full_name; +} +#endif + +#ifdef HAVE_INOTIFY_INIT #if ULTRA_DEBUG == 1 -/* if ultra debugging enabled */ static void -dbg_wdmapPrint(char *msg) +dbg_wdmapPrint(const char *msg) { int i; DBGPRINTF("%s\n", msg); for(i = 0 ; i < nWdmap ; ++i) - DBGPRINTF("wdmap[%d]: wd: %d, file %d, dir %d\n", i, - wdmap[i].wd, wdmap[i].fIdx, wdmap[i].dirIdx); + DBGPRINTF("wdmap[%d]: wd: %d, act %p, name: %s\n", + i, wdmap[i].wd, wdmap[i].act, wdmap[i].act->name); } #endif @@ -391,58 +349,62 @@ finalize_it: RETiRet; } -/* looks up a wdmap entry by filename and returns it's index if found - * or -1 if not found. - */ -static int -wdmapLookupFilename(uchar *pszFileName) -{ - int i = 0; - int wd = -1; - /* Loop through */ - for(i = 0 ; i < nWdmap; ++i) { - if ( wdmap[i].pLstn != NULL && - strcmp((const char*)wdmap[i].pLstn->pszFileName, (const char*)pszFileName) == 0){ - /* Found matching wd */ - wd = wdmap[i].wd; - } - } - - return wd; -} -/* looks up a wdmap entry by pLstn pointer and returns it's index if found - * or -1 if not found. - */ -static int -wdmapLookupListner(lstn_t* pLstn) +/* note: we search backwards, as inotify tends to return increasing wd's */ +static rsRetVal +wdmapAdd(int wd, act_obj_t *const act) { - int i = 0; - int wd = -1; - /* Loop through */ - for(i = 0 ; i < nWdmap; ++i) { - if (wdmap[i].pLstn == pLstn) - wd = wdmap[i].wd; + wd_map_t *newmap; + int newmapsize; + int i; + DEFiRet; + + for(i = nWdmap-1 ; i >= 0 && wdmap[i].wd > wd ; --i) + ; /* just scan */ + if(i >= 0 && wdmap[i].wd == wd) { + LogError(0, RS_RET_INTERNAL_ERROR, "imfile: wd %d already in wdmap!", wd); + ABORT_FINALIZE(RS_RET_FILE_ALREADY_IN_TABLE); + } + ++i; + /* i now points to the entry that is to be moved upwards (or end of map) */ + if(nWdmap == allocMaxWdmap) { + newmapsize = 2 * allocMaxWdmap; + CHKmalloc(newmap = realloc(wdmap, sizeof(wd_map_t) * newmapsize)); + // TODO: handle the error more intelligently? At all possible? -- 2013-10-15 + wdmap = newmap; + allocMaxWdmap = newmapsize; + } + if(i < nWdmap) { + /* we need to shift to make room for new entry */ + memmove(wdmap + i + 1, wdmap + i, sizeof(wd_map_t) * (nWdmap - i)); } + wdmap[i].wd = wd; + wdmap[i].act = act; + ++nWdmap; + DBGPRINTF("add wdmap[%d]: wd %d, act obj %p, path %s\n", i, wd, act, act->name); - return wd; +finalize_it: + RETiRet; } -/* looks up a wdmap entry by dirIdx and returns it's index if found - * or -1 if not found. - */ +/* return wd or -1 on error */ static int -wdmapLookupListnerIdx(const int dirIdx) +in_setupWatch(act_obj_t *const act, const int is_file) { - int i = 0; int wd = -1; - /* Loop through */ - for(i = 0 ; i < nWdmap; ++i) { - if (wdmap[i].dirIdx == dirIdx) - wd = wdmap[i].wd; - } + if(runModConf->opMode != OPMODE_INOTIFY) + goto done; - return wd; + wd = inotify_add_watch(ino_fd, act->name, + (is_file) ? IN_MODIFY : IN_CREATE|IN_DELETE|IN_MOVED_FROM|IN_MOVED_TO); + if(wd < 0) { + LogError(errno, RS_RET_IO_ERROR, "imfile: cannot watch object '%s'", + act->name); + goto done; + } + wdmapAdd(wd, act); + DBGPRINTF("in_setupDirWatch: watch %d added for dir %s(%p)\n", wd, act->name, act); +done: return wd; } /* compare function for bsearch() */ @@ -467,45 +429,6 @@ wdmapLookup(int wd) return bsearch(&wd, wdmap, nWdmap, sizeof(wd_map_t), wdmap_cmp); } -/* note: we search backwards, as inotify tends to return increasing wd's */ -static rsRetVal -wdmapAdd(int wd, const int dirIdx, lstn_t *const pLstn) -{ - wd_map_t *newmap; - int newmapsize; - int i; - DEFiRet; - - for(i = nWdmap-1 ; i >= 0 && wdmap[i].wd > wd ; --i) - ; /* just scan */ - if(i >= 0 && wdmap[i].wd == wd) { - DBGPRINTF("wd %d already in wdmap!\n", wd); - ABORT_FINALIZE(RS_RET_FILE_ALREADY_IN_TABLE); - } - ++i; - /* i now points to the entry that is to be moved upwards (or end of map) */ - if(nWdmap == allocMaxWdmap) { - newmapsize = 2 * allocMaxWdmap; - CHKmalloc(newmap = realloc(wdmap, sizeof(wd_map_t) * newmapsize)); - // TODO: handle the error more intelligently? At all possible? -- 2013-10-15 - wdmap = newmap; - allocMaxWdmap = newmapsize; - } - if(i < nWdmap) { - /* we need to shift to make room for new entry */ - memmove(wdmap + i + 1, wdmap + i, sizeof(wd_map_t) * (nWdmap - i)); - } - wdmap[i].wd = wd; - wdmap[i].dirIdx = dirIdx; - wdmap[i].pLstn = pLstn; - ++nWdmap; - DBGPRINTF("enter into wdmap[%d]: wd %d, dir %d, lstn %s:%s\n",i,wd,dirIdx, - (pLstn == NULL) ? "DIRECTORY" : "FILE", - (pLstn == NULL) ? dirs[dirIdx].dirName : pLstn->pszFileName); - -finalize_it: - RETiRet; -} static rsRetVal wdmapDel(const int wd) @@ -531,145 +454,698 @@ finalize_it: RETiRet; } -#endif /* #if HAVE_INOTIFY_INIT */ - +#endif // #ifdef HAVE_INOTIFY_INIT -/* -* Helper function to combine statefile and workdir -*/ -static int -getFullStateFileName(uchar* pszstatefile, uchar* pszout, int ilenout) +#if defined(OS_SOLARIS) && defined (HAVE_PORT_SOURCE_FILE) +static void ATTR_NONNULL() +fen_setupWatch(act_obj_t *const act) { - int lenout; - const uchar* pszworkdir; + DBGPRINTF("fen_setupWatch: enter, opMode %d\n", runModConf->opMode); + if(runModConf->opMode != OPMODE_FEN) + goto done; - /* Get Raw Workdir, if it is NULL we need to propper handle it */ - pszworkdir = glblGetWorkDirRaw(); + DBGPRINTF("fen_setupWatch: %s\n", act->name); + if(act->pfinf == NULL) { + act->pfinf = malloc(sizeof(struct fileinfo)); + if (act->pfinf == NULL) { + LogError(errno, RS_RET_OUT_OF_MEMORY, "imfile: fen_setupWatch alloc memory " + "for fileinfo failed "); + goto done; + } + if ((act->pfinf->fobj.fo_name = strdup(act->name)) == NULL) { + LogError(errno, RS_RET_OUT_OF_MEMORY, "imfile: fen_setupWatch alloc memory " + "for strdup failed "); + free(act->pfinf); + act->pfinf = NULL; + goto done; + } + act->pfinf->events = FILE_MODIFIED; + act->pfinf->port = glport; + act->bPortAssociated = 0; + } - /* Construct file name */ - lenout = snprintf((char*)pszout, ilenout, "%s/%s", - (char*) (pszworkdir == NULL ? "." : (char*) pszworkdir), (char*)pszstatefile); + DBGPRINTF("fen_setupWatch: bPortAssociated %d\n", act->bPortAssociated); + if(act->bPortAssociated) { + goto done; + } - /* return out length */ - return lenout; -} + struct stat fileInfo; + const int r = stat(act->name, &fileInfo); + if(r == -1) { /* object gone away? */ + DBGPRINTF("fen_setupWatch: file gone away, no watch: '%s'\n", act->name); + goto done; + } + /* note: FEN watch must be re-registered each time - this is what we do now */ + act->pfinf->fobj.fo_atime = fileInfo.st_atim; + act->pfinf->fobj.fo_mtime = fileInfo.st_mtim; + act->pfinf->fobj.fo_ctime = fileInfo.st_ctim; + if(port_associate(glport, PORT_SOURCE_FILE, (uintptr_t)&(act->pfinf->fobj), + act->pfinf->events, (void *)act) == -1) { + LogError(errno, RS_RET_SYS_ERR, "fen_setupWatch: Failed to associate port for file " + ": %s\n", act->pfinf->fobj.fo_name); + goto done; + } else { + /* Port successfull listening now*/ + DBGPRINTF("fen_setupWatch: associated port for file %s\n", act->name); + act->bPortAssociated = 1; + } -/* this generates a state file name suitable for the current file. To avoid - * malloc calls, it must be passed a buffer which should be MAXFNAME large. - * Note: the buffer is not necessarily populated ... always ONLY use the - * RETURN VALUE! - */ -static uchar * ATTR_NONNULL(2) -getStateFileName(lstn_t *const __restrict__ pLstn, - uchar *const __restrict__ buf, - const size_t lenbuf, - const uchar *pszFileName) + DBGPRINTF("in_setupWatch: fen association added for %s\n", act->name); +done: return; +} +#else +static void ATTR_NONNULL() +fen_setupWatch(act_obj_t *const __attribute__((unused)) act) { - uchar *ret; + DBGPRINTF("fen_setupWatch: DUMMY CALLED - not on Solaris?"); +} +#endif /* FEN */ - /* Use pszFileName parameter if set */ - pszFileName = pszFileName == NULL ? pLstn->pszFileName : pszFileName; - - DBGPRINTF("getStateFileName for '%s'\n", pszFileName); - if(pLstn == NULL || pLstn->pszStateFile == NULL) { - snprintf((char*)buf, lenbuf - 1, "imfile-state:%s", pszFileName); - buf[lenbuf-1] = '\0'; /* be on the safe side... */ - uchar *p = buf; - for( ; *p ; ++p) { - if(*p == '/') - *p = '-'; +static void +fs_node_print(const fs_node_t *const node, const int level) +{ + fs_edge_t *chld; + act_obj_t *act; + dbgprintf("node print[%2.2d]: %p edges:\n", level, node); + + for(chld = node->edges ; chld != NULL ; chld = chld->next) { + dbgprintf("node print[%2.2d]: child %p '%s' isFile %d, path: '%s'\n", + level, chld->node, chld->name, chld->is_file, chld->path); + for(int i = 0 ; i < chld->ninst ; ++i) { + dbgprintf("\tinst: %p\n", chld->instarr[i]); + } + for(act = chld->active ; act != NULL ; act = act->next) { + dbgprintf("\tact : %p\n", act); + dbgprintf("\tact : %p: name '%s', wd: %d\n", + act, act->name, act->wd); } - ret = buf; - } else { - ret = pLstn->pszStateFile; } - return ret; + for(chld = node->edges ; chld != NULL ; chld = chld->next) { + fs_node_print(chld->node, level+1); + } } -/* enqueue the read file line as a message. The provided string is - * not freed - this must be done by the caller. +/* add a new file system object if it not yet exists, ignore call + * if it already does. */ -#define MAX_OFFSET_REPRESENTATION_NUM_BYTES 20 -static rsRetVal enqLine(lstn_t *const __restrict__ pLstn, - cstr_t *const __restrict__ cstrLine, - const int64 strtOffs) +static rsRetVal ATTR_NONNULL() +act_obj_add(fs_edge_t *const edge, const char *const name, const int is_file, + const ino_t ino) { + act_obj_t *act; + char basename[MAXFNAME]; DEFiRet; - smsg_t *pMsg; - uchar file_offset[MAX_OFFSET_REPRESENTATION_NUM_BYTES+1]; - const uchar *metadata_names[2] = {(uchar *)"filename",(uchar *)"fileoffset"} ; - const uchar *metadata_values[2] ; - const size_t msgLen = cstrLen(cstrLine); - - if(msgLen == 0) { - /* we do not process empty lines */ - FINALIZE; - } - - CHKiRet(msgConstruct(&pMsg)); - MsgSetFlowControlType(pMsg, eFLOWCTL_FULL_DELAY); - MsgSetInputName(pMsg, pInputName); - if(pLstn->addCeeTag) { - /* Make sure we account for terminating null byte */ - size_t ceeMsgSize = msgLen + CONST_LEN_CEE_COOKIE + 1; - char *ceeMsg; - CHKmalloc(ceeMsg = MALLOC(ceeMsgSize)); - strcpy(ceeMsg, CONST_CEE_COOKIE); - strcat(ceeMsg, (char*)rsCStrGetSzStrNoNULL(cstrLine)); - MsgSetRawMsg(pMsg, ceeMsg, ceeMsgSize); - free(ceeMsg); - } else { - MsgSetRawMsg(pMsg, (char*)rsCStrGetSzStrNoNULL(cstrLine), msgLen); - } - MsgSetMSGoffs(pMsg, 0); /* we do not have a header... */ - MsgSetHOSTNAME(pMsg, glbl.GetLocalHostName(), ustrlen(glbl.GetLocalHostName())); - MsgSetTAG(pMsg, pLstn->pszTag, pLstn->lenTag); - msgSetPRI(pMsg, pLstn->iFacility | pLstn->iSeverity); - MsgSetRuleset(pMsg, pLstn->pRuleset); - if(pLstn->addMetadata) { - metadata_values[0] = pLstn->pszFileName; - snprintf((char *)file_offset, MAX_OFFSET_REPRESENTATION_NUM_BYTES+1, "%lld", strtOffs); - metadata_values[1] = file_offset; - msgAddMultiMetadata(pMsg, metadata_names, metadata_values, 2); + + DBGPRINTF("act_obj_add: edge %p, name '%s'\n", edge, name); + for(act = edge->active ; act != NULL ; act = act->next) { + if(!strcmp(act->name, name)) { + DBGPRINTF("active object '%s' already exists in '%s' - no need to add\n", + name, edge->path); + FINALIZE; + } } - ratelimitAddMsg(pLstn->ratelimiter, &pLstn->multiSub, pMsg); + DBGPRINTF("add new active object '%s' in '%s'\n", name, edge->path); + CHKmalloc(act = calloc(sizeof(act_obj_t), 1)); + CHKmalloc(act->name = strdup(name)); + getBasename((uchar*)basename, (uchar*)name); + CHKmalloc(act->basename = strdup(basename)); + act->edge = edge; + act->ino = ino; + #ifdef HAVE_INOTIFY_INIT + act->wd = in_setupWatch(act, is_file); + #endif + fen_setupWatch(act); + if(is_file) { + const instanceConf_t *const inst = edge->instarr[0];// TODO: same file, multiple instances? + CHKiRet(ratelimitNew(&act->ratelimiter, "imfile", name)); + CHKmalloc(act->multiSub.ppMsgs = MALLOC(inst->nMultiSub * sizeof(smsg_t *))); + act->multiSub.maxElem = inst->nMultiSub; + act->multiSub.nElem = 0; + pollFile(act); + } + + /* all well, add to active list */ + if(edge->active != NULL) { + edge->active->prev = act; + } + act->next = edge->active; + edge->active = act; +//dbgprintf("printout of fs tree after act_obj_add for '%s'\n", name); +//fs_node_print(runModConf->conf_tree, 0); +//dbg_wdmapPrint("wdmap after act_obj_add"); finalize_it: + if(iRet != RS_RET_OK) { + if(act != NULL) { + free(act->name); + free(act); + } + } RETiRet; } -/* try to open a file which has a state file. If the state file does not - * exist or cannot be read, an error is returned. +/* this walks an edges active list and detects and acts on any changes + * seen there. It does NOT detect newly appeared files, as they are not + * inside the active list! */ -static rsRetVal ATTR_NONNULL(1) -openFileWithStateFile(lstn_t *const __restrict__ pLstn) +static void +detect_updates(fs_edge_t *const edge) { - DEFiRet; - strm_t *psSF = NULL; - uchar pszSFNam[MAXFNAME]; - size_t lenSFNam; - struct stat stat_buf; - uchar statefile[MAXFNAME]; - - uchar *const statefn = getStateFileName(pLstn, statefile, sizeof(statefile), NULL); - DBGPRINTF("trying to open state for '%s', state file '%s'\n", - pLstn->pszFileName, statefn); + act_obj_t *act; + struct stat fileInfo; - /* Get full path and file name */ - lenSFNam = getFullStateFileName(statefn, pszSFNam, sizeof(pszSFNam)); + for(act = edge->active ; act != NULL ; ) { + DBGPRINTF("detect_updates checking active obj '%s'\n", act->name); + const int r = stat(act->name, &fileInfo); + if(r == -1) { /* object gone away? */ + DBGPRINTF("object gone away, unlinking: '%s'\n", act->name); + act_obj_t *toDel = act; + act = act->next; + DBGPRINTF("new next act %p\n", act); + act_obj_unlink(toDel); + continue; + } + // TODO: add inode check for change notification! + + /* Note: active nodes may get deleted, so we need to do the + * pointer advancement at the end of the for loop! + */ + act = act->next; + } + + +} + + +/* check if active files need to be processed. This is only needed in + * polling mode. + */ +static void ATTR_NONNULL() +poll_active_files(fs_edge_t *const edge) +{ + if( runModConf->opMode != OPMODE_POLLING + || !edge->is_file + || glbl.GetGlobalInputTermState() != 0) { + return; + } + + act_obj_t *act; + for(act = edge->active ; act != NULL ; act = act->next) { + fen_setupWatch(act); + DBGPRINTF("poll_active_files: polling '%s'\n", act->name); + pollFile(act); + } +} + + +static void ATTR_NONNULL() poll_tree(fs_edge_t *const chld); +static void ATTR_NONNULL() +poll_tree(fs_edge_t *const chld) +{ + struct stat fileInfo; + glob_t files; + DBGPRINTF("poll_tree: chld %p, name '%s', path: %s\n", chld, chld->name, chld->path); + detect_updates(chld); + const int ret = glob((char*)chld->path, runModConf->sortFiles|GLOB_BRACE, NULL, &files); + DBGPRINTF("poll_tree: glob returned %d\n", ret); + if(ret == 0) { + DBGPRINTF("poll_tree: processing %d files\n", (int) files.gl_pathc); + for(unsigned i = 0 ; i < files.gl_pathc ; i++) { + if(glbl.GetGlobalInputTermState() != 0) { + goto done; + } + char *const file = files.gl_pathv[i]; + if(stat(file, &fileInfo) != 0) { + LogError(errno, RS_RET_ERR, + "imfile: poll_tree cannot stat file '%s' - ignored", file); + continue; + } + + const int is_file = S_ISREG(fileInfo.st_mode); + DBGPRINTF("poll_tree: found '%s', File: %d (config file: %d)\n", + file, is_file, chld->is_file); + if(!is_file && S_ISREG(fileInfo.st_mode)) { + LogMsg(0, RS_RET_ERR, LOG_WARNING, + "imfile: '%s' is neither a regular file nor a " + "directory - ignored", file); + continue; + } + if(chld->is_file != is_file) { + LogMsg(0, RS_RET_ERR, LOG_WARNING, + "imfile: '%s' is %s but %s expected - ignored", + file, (is_file) ? "FILE" : "DIRECTORY", + (chld->is_file) ? "FILE" : "DIRECTORY"); + continue; + } + act_obj_add(chld, file, is_file, fileInfo.st_ino); + } + globfree(&files); + } + + poll_active_files(chld); + +done: return; +} + +#ifdef HAVE_INOTIFY_INIT // TODO: shouldn't we use that in polling as well? +static void ATTR_NONNULL() +poll_timeouts(fs_edge_t *const edge) +{ + if(edge->is_file) { + act_obj_t *act; + for(act = edge->active ; act != NULL ; act = act->next) { + if(strmReadMultiLine_isTimedOut(act->pStrm)) { + DBGPRINTF("timeout occured on %s\n", act->name); + pollFile(act); + } + } + } +} +#endif + + +/* destruct a single act_obj object */ +static void +act_obj_destroy(act_obj_t *const act, const int is_deleted) +{ + uchar *statefn; + uchar statefile[MAXFNAME]; + uchar toDel[MAXFNAME]; + + if(act == NULL) + return; + + DBGPRINTF("act_obj_destroy: act %p '%s', wd %d, pStrm %p, is_deleted %d, in_move %d\n", + act, act->name, act->wd, act->pStrm, is_deleted, act->in_move); + if(act->ratelimiter != NULL) { + ratelimitDestruct(act->ratelimiter); + } + if(act->pStrm != NULL) { + const instanceConf_t *const inst = act->edge->instarr[0];// TODO: same file, multiple instances? + pollFile(act); /* get any left-over data */ + if(inst->bRMStateOnDel) { + statefn = getStateFileName(act, statefile, sizeof(statefile)); + getFullStateFileName(statefn, toDel, sizeof(toDel)); + statefn = toDel; + } + persistStrmState(act); + strm.Destruct(&act->pStrm); + /* we delete state file after destruct in case strm obj initiated a write */ + if(is_deleted && !act->in_move && inst->bRMStateOnDel) { + DBGPRINTF("act_obj_destroy: deleting state file %s\n", statefn); + unlink((char*)statefn); + } + } + #ifdef HAVE_INOTIFY_INIT + if(act->wd != -1) { + wdmapDel(act->wd); + } + #endif + #if defined(OS_SOLARIS) && defined (HAVE_PORT_SOURCE_FILE) + if(act->pfinf != NULL) { + free(act->pfinf->fobj.fo_name); + free(act->pfinf); + } + #endif + free(act->basename); + //free(act->statefile); + free(act->multiSub.ppMsgs); + #if defined(OS_SOLARIS) && defined (HAVE_PORT_SOURCE_FILE) + act->is_deleted = 1; + #else + free(act->name); + free(act); + #endif +} + + +/* destroy complete act list starting at given node */ +static void +act_obj_destroy_all(act_obj_t *act) +{ + if(act == NULL) + return; + + DBGPRINTF("act_obj_destroy_all: act %p '%s', wd %d, pStrm %p\n", act, act->name, act->wd, act->pStrm); + while(act != NULL) { + act_obj_t *const toDel = act; + act = act->next; + act_obj_destroy(toDel, 0); + } +} + +#if 0 +/* debug: find if ptr is still present in list */ +static void +chk_active(const act_obj_t *act, const act_obj_t *const deleted) +{ + while(act != NULL) { + DBGPRINTF("chk_active %p vs %p\n", act, deleted); + if(act->prev == deleted) + DBGPRINTF("chk_active %p prev points to %p\n", act, deleted); + if(act->next == deleted) + DBGPRINTF("chk_active %p next points to %p\n", act, deleted); + act = act->next; + DBGPRINTF("chk_active next %p\n", act); + } +} +#endif + +/* unlink act object from linked list and then + * destruct it. + */ +static void //ATTR_NONNULL() +act_obj_unlink(act_obj_t *const act) +{ + DBGPRINTF("act_obj_unlink %p: %s\n", act, act->name); + if(act->prev == NULL) { + act->edge->active = act->next; + } else { + act->prev->next = act->next; + } + if(act->next != NULL) { + act->next->prev = act->prev; + } + act_obj_destroy(act, 1); +//dbgprintf("printout of fs tree post unlink\n"); +//fs_node_print(runModConf->conf_tree, 0); +//dbg_wdmapPrint("wdmap after"); +} + +static void +fs_node_destroy(fs_node_t *const node) +{ + fs_edge_t *edge; + DBGPRINTF("node destroy: %p edges:\n", node); + + for(edge = node->edges ; edge != NULL ; ) { + fs_node_destroy(edge->node); + fs_edge_t *const toDel = edge; + edge = edge->next; + act_obj_destroy_all(toDel->active); + free(toDel->name); + free(toDel->path); + free(toDel->instarr); + free(toDel); + } + free(node); +} + +static void ATTR_NONNULL(1, 2) +fs_node_walk(fs_node_t *const node, + void (*f_usr)(fs_edge_t*const)) +{ + DBGPRINTF("node walk: %p edges:\n", node); + + fs_edge_t *edge; + for(edge = node->edges ; edge != NULL ; edge = edge->next) { + DBGPRINTF("node walk: child %p '%s'\n", edge->node, edge->name); + f_usr(edge); + fs_node_walk(edge->node, f_usr); + } +} + + + +/* add a file system object to config tree (or update existing node with new monitor) + */ +static rsRetVal +fs_node_add(fs_node_t *const node, + const uchar *const toFind, + const size_t pathIdx, + instanceConf_t *const inst) +{ + DEFiRet; + fs_edge_t *newchld = NULL; + int i; + + DBGPRINTF("fs_node_add(%p, '%s') enter, idx %zd\n", + node, toFind+pathIdx, pathIdx); + assert(toFind[0] != '\0'); + for(i = pathIdx ; (toFind[i] != '\0') && (toFind[i] != '/') ; ++i) + /*JUST SKIP*/; + const int isFile = (toFind[i] == '\0') ? 1 : 0; + uchar ourPath[PATH_MAX]; + if(i == 0) { + ourPath[0] = '/'; + ourPath[1] = '\0'; + } else { + memcpy(ourPath, toFind, i); + ourPath[i] = '\0'; + } + const size_t nextPathIdx = i+1; + const size_t len = i - pathIdx; + uchar name[PATH_MAX]; + memcpy(name, toFind+pathIdx, len); + name[len] = '\0'; + DBGPRINTF("fs_node_add: name '%s'\n", name); + + fs_edge_t *chld; + for(chld = node->edges ; chld != NULL ; chld = chld->next) { + if(!ustrcmp(chld->name, name)) { + DBGPRINTF("fs_node_add(%p, '%s') found '%s'\n", chld->node, toFind, name); + /* add new instance */ + chld->ninst++; + CHKmalloc(chld->instarr = realloc(chld->instarr, sizeof(instanceConf_t*) * chld->ninst)); + chld->instarr[chld->ninst-1] = inst; + /* recurse */ + if(!isFile) { + CHKiRet(fs_node_add(chld->node, toFind, nextPathIdx, inst)); + } + FINALIZE; + } + } + + /* could not find node --> add it */ + DBGPRINTF("fs_node_add(%p, '%s') did not find '%s' - adding it\n", + node, toFind, name); + CHKmalloc(newchld = calloc(sizeof(fs_edge_t), 1)); + CHKmalloc(newchld->name = ustrdup(name)); + CHKmalloc(newchld->node = calloc(sizeof(fs_node_t), 1)); + CHKmalloc(newchld->path = ustrdup(ourPath)); + CHKmalloc(newchld->instarr = calloc(sizeof(instanceConf_t*), 1)); + newchld->instarr[0] = inst; + newchld->is_file = isFile; + newchld->ninst = 1; + newchld->parent = node; + + DBGPRINTF("fs_node_add(%p, '%s') returns %p\n", node, toFind, newchld->node); + + if(!isFile) { + CHKiRet(fs_node_add(newchld->node, toFind, nextPathIdx, inst)); + } + + /* link to list */ + newchld->next = node->edges; + node->edges = newchld; +finalize_it: + if(iRet != RS_RET_OK) { + if(newchld != NULL) { + free(newchld->name); + free(newchld->node); + free(newchld->path); + free(newchld->instarr); + free(newchld); + } + } + RETiRet; +} + +#if 0 //TODO: check if we need (specialised?) versions of this? +/* we receive a notification that a new object is found *beneath* + * act. This function now finds the right spot to place it and the + * activate the monitor. + * TODO: think if it is worth optimizing this based on the inotify-provided + * name. But it's complex in any case... + */ +static rsRetVal ATTR_NONNULL(1, 2) +fs_node_notify_new_obj(act_obj_t *const act, const char *const name) +{ + DBGPRINTF("fs_node_notify_new_obj: act->name '%s', name '%s'\n", + act->name, name); +#if 0 + char fullname[MAXFNAME]; + snprintf(fullname, MAXFNAME, "%s/%s", act->name, name); +// act_obj_add(act->edge->node, fullname, 0); +#endif + fs_node_walk(act->edge->node, poll_tree); + return RS_RET_OK; +} + +static rsRetVal ATTR_NONNULL(1, 2) +fs_node_notify_file_del(act_obj_t *const act, const char *const name) +{ + DBGPRINTF("fs_node_notify_file_del: act->name '%s', name '%s'\n", + act->name, name); + fs_node_walk(act->edge->parent, poll_tree); + // TODO: 1. impl: walk tree, 2. impl: use inotify name + return RS_RET_OK; +} +#endif + + +/* Helper function to combine statefile and workdir + * This function is guranteed to work only on config data and DOES NOT + * open or otherwise modify disk file state. + */ +static int ATTR_NONNULL() +getFullStateFileName(const uchar *const pszstatefile, uchar *const pszout, const size_t ilenout) +{ + int lenout; + const uchar* pszworkdir; + + /* Get Raw Workdir, if it is NULL we need to propper handle it */ + pszworkdir = glblGetWorkDirRaw(); + + /* Construct file name */ + lenout = snprintf((char*)pszout, ilenout, "%s/%s", + (char*) (pszworkdir == NULL ? "." : (char*) pszworkdir), (char*)pszstatefile); + + /* return out length */ + return lenout; +} + + +#if 0 +/* generate a state file name for the given file name + * the file is stored in given buf, which must be of MAXFNAME length. + */ +static void ATTR_NONNULL(1, 2) +genStateFileName(const char *const name, + uchar *const __restrict__ buf, + const size_t lenbuf) +{ + DBGPRINTF("genStateFileName for '%s'\n", name); + snprintf((char*)buf, lenbuf - 1, "imfile-state:%s", name); + buf[lenbuf-1] = '\0'; /* be on the safe side... */ + uchar *p = buf; + for( ; *p ; ++p) { + if(*p == '/') + *p = '-'; + } +} +#endif + + +/* this generates a state file name suitable for the given file. To avoid + * malloc calls, it must be passed a buffer which should be MAXFNAME large. + * Note: the buffer is not necessarily populated ... always ONLY use the + * RETURN VALUE! + * This function is guranteed to work only on config data and DOES NOT + * open or otherwise modify disk file state. + */ +static uchar * ATTR_NONNULL(1, 2) +getStateFileName(const act_obj_t *const act, + uchar *const __restrict__ buf, + const size_t lenbuf) +{ + DBGPRINTF("getStateFileName for '%s'\n", act->name); + snprintf((char*)buf, lenbuf - 1, "imfile-state:%lld", (long long) act->ino); + DBGPRINTF("getStateFileName: stat file name now is %s\n", buf); + return buf; +#if 0 + uchar *ret; + //if(inst->pszStateFile == NULL) { + genStateFileName(act->name, buf, lenbuf); + ret = buf; + //} else { + //ret = inst->pszStateFile; + //} + return ret; +#endif +} + + +/* enqueue the read file line as a message. The provided string is + * not freed - this must be done by the caller. + */ +#define MAX_OFFSET_REPRESENTATION_NUM_BYTES 20 +static rsRetVal ATTR_NONNULL(1,2) +enqLine(act_obj_t *const act, + cstr_t *const __restrict__ cstrLine, + const int64 strtOffs) +{ + DEFiRet; + const instanceConf_t *const inst = act->edge->instarr[0];// TODO: same file, multiple instances? + smsg_t *pMsg; + uchar file_offset[MAX_OFFSET_REPRESENTATION_NUM_BYTES+1]; + const uchar *metadata_names[2] = {(uchar *)"filename",(uchar *)"fileoffset"} ; + const uchar *metadata_values[2] ; + const size_t msgLen = cstrLen(cstrLine); + + if(msgLen == 0) { + /* we do not process empty lines */ + FINALIZE; + } + + CHKiRet(msgConstruct(&pMsg)); + MsgSetFlowControlType(pMsg, eFLOWCTL_FULL_DELAY); + MsgSetInputName(pMsg, pInputName); + if(inst->addCeeTag) { + /* Make sure we account for terminating null byte */ + size_t ceeMsgSize = msgLen + CONST_LEN_CEE_COOKIE + 1; + char *ceeMsg; + CHKmalloc(ceeMsg = MALLOC(ceeMsgSize)); + strcpy(ceeMsg, CONST_CEE_COOKIE); + strcat(ceeMsg, (char*)rsCStrGetSzStrNoNULL(cstrLine)); + MsgSetRawMsg(pMsg, ceeMsg, ceeMsgSize); + free(ceeMsg); + } else { + MsgSetRawMsg(pMsg, (char*)rsCStrGetSzStrNoNULL(cstrLine), msgLen); + } + MsgSetMSGoffs(pMsg, 0); /* we do not have a header... */ + MsgSetHOSTNAME(pMsg, glbl.GetLocalHostName(), ustrlen(glbl.GetLocalHostName())); + MsgSetTAG(pMsg, inst->pszTag, inst->lenTag); + msgSetPRI(pMsg, inst->iFacility | inst->iSeverity); + MsgSetRuleset(pMsg, inst->pBindRuleset); + if(inst->addMetadata) { + metadata_values[0] = (const uchar*)act->name; + snprintf((char *)file_offset, MAX_OFFSET_REPRESENTATION_NUM_BYTES+1, "%lld", strtOffs); + metadata_values[1] = file_offset; + msgAddMultiMetadata(pMsg, metadata_names, metadata_values, 2); + } + ratelimitAddMsg(act->ratelimiter, &act->multiSub, pMsg); +finalize_it: + RETiRet; +} + + +/* try to open a file which has a state file. If the state file does not + * exist or cannot be read, an error is returned. + */ +static rsRetVal ATTR_NONNULL(1) +openFileWithStateFile(act_obj_t *const act) +{ + DEFiRet; + strm_t *psSF = NULL; + uchar pszSFNam[MAXFNAME]; + size_t lenSFNam; + struct stat stat_buf; + uchar statefile[MAXFNAME]; + const instanceConf_t *const inst = act->edge->instarr[0];// TODO: same file, multiple instances? + + uchar *const statefn = getStateFileName(act, statefile, sizeof(statefile)); + DBGPRINTF("trying to open state for '%s', state file '%s'\n", act->name, statefn); + + /* persist state file base name in act, as this may be used for MOVE operations */ + //free(act->statefile); + //CHKmalloc(act->statefile = strdup((char*) statefile)); + + /* Get full path and file name */ + lenSFNam = getFullStateFileName(statefn, pszSFNam, sizeof(pszSFNam)); /* check if the file exists */ if(stat((char*) pszSFNam, &stat_buf) == -1) { if(errno == ENOENT) { - DBGPRINTF("NO state file (%s) exists for '%s'\n", pszSFNam, pLstn->pszFileName); + DBGPRINTF("NO state file (%s) exists for '%s'\n", pszSFNam, act->name); ABORT_FINALIZE(RS_RET_FILE_NOT_FOUND); } else { char errStr[1024]; rs_strerror_r(errno, errStr, sizeof(errStr)); DBGPRINTF("error trying to access state file for '%s':%s\n", - pLstn->pszFileName, errStr); + act->name, errStr); ABORT_FINALIZE(RS_RET_IO_ERROR); } } @@ -680,39 +1156,27 @@ openFileWithStateFile(lstn_t *const __restrict__ pLstn) CHKiRet(strm.SettOperationsMode(psSF, STREAMMODE_READ)); CHKiRet(strm.SetsType(psSF, STREAMTYPE_FILE_SINGLE)); CHKiRet(strm.SetFName(psSF, pszSFNam, lenSFNam)); - CHKiRet(strm.SetFileNotFoundError(psSF, pLstn->fileNotFoundError)); + CHKiRet(strm.SetFileNotFoundError(psSF, inst->fileNotFoundError)); CHKiRet(strm.ConstructFinalize(psSF)); /* read back in the object */ - CHKiRet(obj.Deserialize(&pLstn->pStrm, (uchar*) "strm", psSF, NULL, pLstn)); + CHKiRet(obj.Deserialize(&act->pStrm, (uchar*) "strm", psSF, NULL, act)); DBGPRINTF("deserialized state file, state file base name '%s', " - "configured base name '%s'\n", pLstn->pStrm->pszFName, - pLstn->pszFileName); - if(ustrcmp(pLstn->pStrm->pszFName, pLstn->pszFileName)) { - if (pLstn->masterLstn != NULL) { - /* StateFile was migrated from a filemove. - We need to correct the stream Filename and continue */ - CHKmalloc(pLstn->pStrm->pszFName = ustrdup(pLstn->pszFileName)); - DBGPRINTF("statefile was migrated from a file rename for '%s'\n", pLstn->pszFileName); - } else { - LogError(0, RS_RET_STATEFILE_WRONG_FNAME, "imfile: state file '%s' " - "contains file name '%s', but is used for file '%s'. State " - "file deleted, starting from begin of file.", - pszSFNam, pLstn->pStrm->pszFName, pLstn->pszFileName); + "configured base name '%s'\n", act->pStrm->pszFName, + act->name); + /* correct file name (if renamed) TODO: refactor state file! */ + free(act->pStrm->pszFName); + CHKmalloc(act->pStrm->pszFName = ustrdup(act->name)); - unlink((char*)pszSFNam); - ABORT_FINALIZE(RS_RET_STATEFILE_WRONG_FNAME); - } - } - - strm.CheckFileChange(pLstn->pStrm); - CHKiRet(strm.SeekCurrOffs(pLstn->pStrm)); + strm.CheckFileChange(act->pStrm); + CHKiRet(strm.SeekCurrOffs(act->pStrm)); /* note: we do not delete the state file, so that the last position remains * known even in the case that rsyslogd aborts for some reason (like powerfail) */ finalize_it: + dbgprintf("state file out %p\n", psSF); if(psSF != NULL) strm.Destruct(&psSF); @@ -724,28 +1188,31 @@ finalize_it: * checked before calling it. */ static rsRetVal -openFileWithoutStateFile(lstn_t *const __restrict__ pLstn) +openFileWithoutStateFile(act_obj_t *const act) { DEFiRet; struct stat stat_buf; - DBGPRINTF("clean startup withOUT state file for '%s'\n", pLstn->pszFileName); - if(pLstn->pStrm != NULL) - strm.Destruct(&pLstn->pStrm); - CHKiRet(strm.Construct(&pLstn->pStrm)); - CHKiRet(strm.SettOperationsMode(pLstn->pStrm, STREAMMODE_READ)); - CHKiRet(strm.SetsType(pLstn->pStrm, STREAMTYPE_FILE_MONITOR)); - CHKiRet(strm.SetFName(pLstn->pStrm, pLstn->pszFileName, strlen((char*) pLstn->pszFileName))); - CHKiRet(strm.SetFileNotFoundError(pLstn->pStrm, pLstn->fileNotFoundError)); - CHKiRet(strm.ConstructFinalize(pLstn->pStrm)); + // TODO: same file, multiple instances? + const instanceConf_t *const inst = act->edge->instarr[0]; + + DBGPRINTF("clean startup withOUT state file for '%s'\n", act->name); + if(act->pStrm != NULL) + strm.Destruct(&act->pStrm); + CHKiRet(strm.Construct(&act->pStrm)); + CHKiRet(strm.SettOperationsMode(act->pStrm, STREAMMODE_READ)); + CHKiRet(strm.SetsType(act->pStrm, STREAMTYPE_FILE_MONITOR)); + CHKiRet(strm.SetFName(act->pStrm, (uchar*)act->name, strlen(act->name))); + CHKiRet(strm.SetFileNotFoundError(act->pStrm, inst->fileNotFoundError)); + CHKiRet(strm.ConstructFinalize(act->pStrm)); /* As a state file not exist, this is a fresh start. seek to file end * when freshStartTail is on. */ - if(pLstn->freshStartTail){ - if(stat((char*) pLstn->pszFileName, &stat_buf) != -1) { - pLstn->pStrm->iCurrOffs = stat_buf.st_size; - CHKiRet(strm.SeekCurrOffs(pLstn->pStrm)); + if(inst->freshStartTail){ + if(stat((char*) act->name, &stat_buf) != -1) { + act->pStrm->iCurrOffs = stat_buf.st_size; + CHKiRet(strm.SeekCurrOffs(act->pStrm)); } } @@ -757,18 +1224,18 @@ finalize_it: * if so, reading it in. Processing continues from the last known location. */ static rsRetVal -openFile(lstn_t *const __restrict__ pLstn) +openFile(act_obj_t *const act) { DEFiRet; + const instanceConf_t *const inst = act->edge->instarr[0];// TODO: same file, multiple instances? - CHKiRet_Hdlr(openFileWithStateFile(pLstn)) { - CHKiRet(openFileWithoutStateFile(pLstn)); + CHKiRet_Hdlr(openFileWithStateFile(act)) { + CHKiRet(openFileWithoutStateFile(act)); } - DBGPRINTF("breopenOnTruncate %d for '%s'\n", - pLstn->reopenOnTruncate, pLstn->pszFileName); - CHKiRet(strm.SetbReopenOnTruncate(pLstn->pStrm, pLstn->reopenOnTruncate)); - strmSetReadTimeout(pLstn->pStrm, pLstn->readTimeout); + DBGPRINTF("breopenOnTruncate %d for '%s'\n", inst->reopenOnTruncate, act->name); + CHKiRet(strm.SetbReopenOnTruncate(act->pStrm, inst->reopenOnTruncate)); + strmSetReadTimeout(act->pStrm, inst->readTimeout); finalize_it: RETiRet; @@ -789,41 +1256,46 @@ static void pollFileCancelCleanup(void *pArg) /* pollFile needs to be split due to the unfortunate pthread_cancel_push() macros. */ -static rsRetVal ATTR_NONNULL(1, 3) -pollFileReal(lstn_t *pLstn, int *pbHadFileData, cstr_t **pCStr) +static rsRetVal ATTR_NONNULL() +pollFileReal(act_obj_t *act, cstr_t **pCStr) { int64 strtOffs; DEFiRet; - int nProcessed = 0; - if(pLstn->pStrm == NULL) { - CHKiRet(openFile(pLstn)); /* open file */ + + DBGPRINTF("pollFileReal enter, pStrm %p, name '%s'\n", act->pStrm, act->name); + DBGPRINTF("pollFileReal enter, edge %p\n", act->edge); + DBGPRINTF("pollFileReal enter, edge->instarr %p\n", act->edge->instarr); + + instanceConf_t *const inst = act->edge->instarr[0];// TODO: same file, multiple instances? + + if(act->pStrm == NULL) { + CHKiRet(openFile(act)); /* open file */ } /* loop below will be exited when strmReadLine() returns EOF */ while(glbl.GetGlobalInputTermState() == 0) { - if(pLstn->maxLinesAtOnce != 0 && nProcessed >= pLstn->maxLinesAtOnce) + if(inst->maxLinesAtOnce != 0 && nProcessed >= inst->maxLinesAtOnce) break; - if(pLstn->startRegex == NULL) { - CHKiRet(strm.ReadLine(pLstn->pStrm, pCStr, pLstn->readMode, pLstn->escapeLF, - pLstn->trimLineOverBytes, &strtOffs)); + if(inst->startRegex == NULL) { + CHKiRet(strm.ReadLine(act->pStrm, pCStr, inst->readMode, inst->escapeLF, + inst->trimLineOverBytes, &strtOffs)); } else { - CHKiRet(strmReadMultiLine(pLstn->pStrm, pCStr, &pLstn->end_preg, - pLstn->escapeLF, pLstn->discardTruncatedMsg, pLstn->msgDiscardingError, &strtOffs)); + CHKiRet(strmReadMultiLine(act->pStrm, pCStr, &inst->end_preg, + inst->escapeLF, inst->discardTruncatedMsg, inst->msgDiscardingError, &strtOffs)); } ++nProcessed; - if(pbHadFileData != NULL) - *pbHadFileData = 1; /* this is just a flag, so set it and forget it */ - CHKiRet(enqLine(pLstn, *pCStr, strtOffs)); /* process line */ + runModConf->bHadFileData = 1; /* this is just a flag, so set it and forget it */ + CHKiRet(enqLine(act, *pCStr, strtOffs)); /* process line */ rsCStrDestruct(pCStr); /* discard string (must be done by us!) */ - if(pLstn->iPersistStateInterval > 0 && ++pLstn->nRecords >= pLstn->iPersistStateInterval) { - persistStrmState(pLstn); - pLstn->nRecords = 0; + if(inst->iPersistStateInterval > 0 && ++act->nRecords >= inst->iPersistStateInterval) { + persistStrmState(act); + act->nRecords = 0; } } finalize_it: - multiSubmitFlush(&pLstn->multiSub); + multiSubmitFlush(&act->multiSub); if(*pCStr != NULL) { rsCStrDestruct(pCStr); @@ -834,7 +1306,7 @@ finalize_it: /* poll a file, need to check file rollover etc. open file if not open */ static rsRetVal ATTR_NONNULL(1) -pollFile(lstn_t *pLstn, int *pbHadFileData) +pollFile(act_obj_t *const act) { cstr_t *pCStr = NULL; DEFiRet; @@ -842,7 +1314,7 @@ pollFile(lstn_t *pLstn, int *pbHadFileData) * otherwise do not work if I include the _cleanup_pop() inside an if... -- rgerhards, 2008-08-14 */ pthread_cleanup_push(pollFileCancelCleanup, &pCStr); - iRet = pollFileReal(pLstn, pbHadFileData, &pCStr); + iRet = pollFileReal(act, &pCStr); pthread_cleanup_pop(0); RETiRet; } @@ -926,18 +1398,11 @@ getBasename(uchar *const __restrict__ basen, uchar *const __restrict__ path) } /* this function checks instance parameters and does some required pre-processing - * (e.g. split filename in path and actual name) - * Note: we do NOT use dirname()/basename() as they have portability problems. */ static rsRetVal ATTR_NONNULL() -checkInstance(instanceConf_t *inst) +checkInstance(instanceConf_t *const inst) { - char dirn[MAXFNAME]; - uchar basen[MAXFNAME]; - int i; - struct stat sb; - int r; - sbool hasWildcard; + uchar curr_wd[MAXFNAME]; DEFiRet; /* this is primarily for the clang static analyzer, but also @@ -946,47 +1411,36 @@ checkInstance(instanceConf_t *inst) if(inst->pszFileName == NULL) ABORT_FINALIZE(RS_RET_INTERNAL_ERROR); - i = getBasename(basen, inst->pszFileName); - if (i == -1) { - LogError(0, RS_RET_CONFIG_ERROR, "imfile: file path '%s' does not include a " - "basename component", inst->pszFileName); - ABORT_FINALIZE(RS_RET_CONFIG_ERROR); - } - - memcpy(dirn, inst->pszFileName, i); /* do not copy slash */ - dirn[i] = '\0'; - - DBGPRINTF("checking instance for directory [%s]\n", dirn); - - CHKmalloc(inst->pszFileBaseName = ustrdup(basen)); - CHKmalloc(inst->pszDirName = ustrdup(dirn)); + if(loadModConf->normalizePath) { + if(inst->pszFileName[0] == '.' && inst->pszFileName[1] == '/') { + DBGPRINTF("imfile: removing heading './' from name '%s'\n", inst->pszFileName); + memmove(inst->pszFileName, inst->pszFileName+2, ustrlen(inst->pszFileName) - 1); + } - if(dirn[0] == '\0') { - dirn[0] = '/'; - dirn[1] = '\0'; + if(inst->pszFileName[0] != '/') { + if(getcwd((char*)curr_wd, MAXFNAME) == NULL || curr_wd[0] != '/') { + LogError(errno, RS_RET_ERR, "imfile: error querying current working " + "directory - can not continue with %s", inst->pszFileName); + ABORT_FINALIZE(RS_RET_ERR); + } + const size_t len_curr_wd = ustrlen(curr_wd); + if(len_curr_wd + ustrlen(inst->pszFileName) + 1 >= MAXFNAME) { + LogError(0, RS_RET_ERR, "imfile: length of configured file and current " + "working directory exceeds permitted size - ignoring %s", + inst->pszFileName); + ABORT_FINALIZE(RS_RET_ERR); + } + curr_wd[len_curr_wd] = '/'; + strcpy((char*)curr_wd+len_curr_wd+1, (char*)inst->pszFileName); + free(inst->pszFileName); + CHKmalloc(inst->pszFileName = ustrdup(curr_wd)); + } } + dbgprintf("imfile: adding file monitor for '%s'\n", inst->pszFileName); - /* Check for Wildcards in FileBaseName. - * If there is one, we need to truncate before the wildcard in order - * to proceed further tests. - */ - hasWildcard = containsGlobWildcard(dirn); - if(hasWildcard) { - DBGPRINTF("wildcard in dirname, do not check if dir exists for [%s]\n", dirn); - FINALIZE + if(inst->pszTag != NULL) { + inst->lenTag = ustrlen(inst->pszTag); } - - r = stat(dirn, &sb); - if(r != 0) { - LogError(errno, RS_RET_CONFIG_ERROR, "imfile warning: directory '%s'", dirn); - ABORT_FINALIZE(RS_RET_CONFIG_ERROR); - } - if(!S_ISDIR(sb.st_mode)) { - LogError(0, RS_RET_CONFIG_ERROR, "imfile warning: configured directory " - "'%s' is NOT a directory", dirn); - ABORT_FINALIZE(RS_RET_CONFIG_ERROR); - } - finalize_it: RETiRet; } @@ -1056,165 +1510,6 @@ finalize_it: } -/* This adds a new listener object to the bottom of the list, but - * it does NOT initialize any data members except for the list - * pointers themselves. - */ -static rsRetVal ATTR_NONNULL() -lstnAdd(lstn_t **const newLstn) -{ - lstn_t *pLstn; - DEFiRet; - - CHKmalloc(pLstn = (lstn_t*) MALLOC(sizeof(lstn_t))); - if(runModConf->pRootLstn == NULL) { - runModConf->pRootLstn = pLstn; - pLstn->prev = NULL; - } else { - runModConf->pTailLstn->next = pLstn; - pLstn->prev = runModConf->pTailLstn; - } - runModConf->pTailLstn = pLstn; - pLstn->next = NULL; - *newLstn = pLstn; - -finalize_it: - RETiRet; -} - -/* delete a listener object */ -static void ATTR_NONNULL(1) -lstnDel(lstn_t *pLstn) -{ - DBGPRINTF("lstnDel called for %s\n", pLstn->pszFileName); - if(pLstn->pStrm != NULL) { /* stream open? */ - persistStrmState(pLstn); - strm.Destruct(&(pLstn->pStrm)); - } - ratelimitDestruct(pLstn->ratelimiter); - free(pLstn->multiSub.ppMsgs); - free(pLstn->pszFileName); - free(pLstn->pszTag); - free(pLstn->pszStateFile); - free(pLstn->pszBaseName); - if(pLstn->startRegex != NULL) - regfree(&pLstn->end_preg); -#ifdef HAVE_INOTIFY_INIT - if (pLstn->movedfrom_statefile != NULL) - free(pLstn->movedfrom_statefile); -#endif -#if defined(OS_SOLARIS) && defined (HAVE_PORT_SOURCE_FILE) - if (pLstn->pfinf != NULL) { - free(pLstn->pfinf->fobj.fo_name); - free(pLstn->pfinf); - } -#endif - if(pLstn == runModConf->pRootLstn) - runModConf->pRootLstn = pLstn->next; - if(pLstn == runModConf->pTailLstn) - runModConf->pTailLstn = pLstn->prev; - if(pLstn->next != NULL) - pLstn->next->prev = pLstn->prev; - if(pLstn->prev != NULL) - pLstn->prev->next = pLstn->next; - free(pLstn); -} - -/* This function is called when a new listener shall be added. - * It also does some late stage error checking on the config - * and reports issues it finds. - */ -static rsRetVal ATTR_NONNULL(1) -addListner(instanceConf_t *const inst) -{ - DEFiRet; - lstn_t *pThis; - sbool hasWildcard; - - hasWildcard = containsGlobWildcard((char*)inst->pszFileBaseName); - DBGPRINTF("addListner file '%s', wildcard detected: %s\n", - inst->pszFileBaseName, (hasWildcard ? "TRUE" : "FALSE")); - - if(hasWildcard) { - if(runModConf->opMode == OPMODE_POLLING) { - LogError(0, RS_RET_IMFILE_WILDCARD, - "imfile: The to-be-monitored file \"%s\" contains " - "wildcards. This is not supported in " - "polling mode.", inst->pszFileName); - ABORT_FINALIZE(RS_RET_IMFILE_WILDCARD); - } else if(inst->pszStateFile != NULL) { - LogError(0, RS_RET_IMFILE_WILDCARD, - "imfile: warning: it looks like to-be-monitored " - "file \"%s\" contains wildcards. This usually " - "does not work well with specifying a state file.", - inst->pszFileName); - } - } - - CHKiRet(lstnAdd(&pThis)); - pThis->hasWildcard = hasWildcard; - pThis->pszDirName = inst->pszDirName; - CHKmalloc(pThis->pszFileName = ustrdup(inst->pszFileName)); - CHKmalloc(pThis->pszBaseName = ustrdup(inst->pszFileBaseName)); /* be consistent with expanded wildcards! */ - CHKmalloc(pThis->pszTag = ustrdup(inst->pszTag)); - pThis->lenTag = ustrlen(pThis->pszTag); - if(inst->pszStateFile == NULL) { - pThis->pszStateFile = NULL; - } else { - pThis->pszStateFile = ustrdup(inst->pszStateFile); - } - - CHKiRet(ratelimitNew(&pThis->ratelimiter, "imfile", (char*)inst->pszFileName)); - CHKmalloc(pThis->multiSub.ppMsgs = MALLOC(inst->nMultiSub * sizeof(smsg_t *))); - pThis->multiSub.maxElem = inst->nMultiSub; - pThis->multiSub.nElem = 0; - pThis->iSeverity = inst->iSeverity; - pThis->iFacility = inst->iFacility; - pThis->maxLinesAtOnce = inst->maxLinesAtOnce; - pThis->trimLineOverBytes = inst->trimLineOverBytes; - pThis->iPersistStateInterval = inst->iPersistStateInterval; - pThis->readMode = inst->readMode; - pThis->startRegex = inst->startRegex; /* no strdup, as it is read-only */ - if(pThis->startRegex != NULL) { - const int errcode = regcomp(&pThis->end_preg, (char*)pThis->startRegex, REG_EXTENDED); - if(errcode != 0) { - char errbuff[512]; - regerror(errcode, &pThis->end_preg, errbuff, sizeof(errbuff)); - LogError(0, NO_ERRCODE, "imfile: %s\n", errbuff); - ABORT_FINALIZE(RS_RET_ERR); - } - } - pThis->discardTruncatedMsg = inst->discardTruncatedMsg; - pThis->msgDiscardingError = inst->msgDiscardingError; - pThis->bRMStateOnDel = inst->bRMStateOnDel; - pThis->escapeLF = inst->escapeLF; - pThis->reopenOnTruncate = inst->reopenOnTruncate; - pThis->addMetadata = (inst->addMetadata == ADD_METADATA_UNSPECIFIED) ? - hasWildcard : inst->addMetadata; - pThis->addCeeTag = inst->addCeeTag; - pThis->readTimeout = inst->readTimeout; - pThis->freshStartTail = inst->freshStartTail; - pThis->fileNotFoundError = inst->fileNotFoundError; - pThis->pRuleset = inst->pBindRuleset; - pThis->nRecords = 0; - pThis->pStrm = NULL; - pThis->prevLineSegment = NULL; - pThis->masterLstn = NULL; /* we *are* a master! */ - #ifdef HAVE_INOTIFY_INIT - /* Init Moved Files variables (Used for MOVED_TO/MOVED_FROM)*/ - pThis->movedfrom_statefile = NULL; - pThis->movedfrom_cookie = 0; - #endif - #if defined(OS_SOLARIS) && defined (HAVE_PORT_SOURCE_FILE) - pThis->pfinf = NULL; - pThis->bPortAssociated = 0; - #endif - -finalize_it: - RETiRet; -} - - BEGINnewInpInst struct cnfparamvals *pvals; instanceConf_t *inst; @@ -1242,7 +1537,7 @@ CODESTARTnewInpInst } else if(!strcmp(inppblk.descr[i].name, "statefile")) { inst->pszStateFile = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(inppblk.descr[i].name, "removestateondelete")) { - inst->bRMStateOnDel = (uint8_t) pvals[i].val.d.n; + inst->bRMStateOnDel = (uint8_t) pvals[i].val.d.n; // TODO: duplicate! } else if(!strcmp(inppblk.descr[i].name, "tag")) { inst->pszTag = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(inppblk.descr[i].name, "ruleset")) { @@ -1260,7 +1555,7 @@ CODESTARTnewInpInst } else if(!strcmp(inppblk.descr[i].name, "msgdiscardingerror")) { inst->msgDiscardingError = (sbool) pvals[i].val.d.n; } else if(!strcmp(inppblk.descr[i].name, "deletestateonfiledelete")) { - inst->bRMStateOnDel = (sbool) pvals[i].val.d.n; + inst->bRMStateOnDel = (sbool) pvals[i].val.d.n; // TODO: duplicate! } else if(!strcmp(inppblk.descr[i].name, "addmetadata")) { inst->addMetadata = (sbool) pvals[i].val.d.n; } else if (!strcmp(inppblk.descr[i].name, "addceetag")) { @@ -1301,6 +1596,16 @@ CODESTARTnewInpInst "at the same time --- remove one of them"); ABORT_FINALIZE(RS_RET_PARAM_NOT_PERMITTED); } + + if(inst->startRegex != NULL) { + const int errcode = regcomp(&inst->end_preg, (char*)inst->startRegex, REG_EXTENDED); + if(errcode != 0) { + char errbuff[512]; + regerror(errcode, &inst->end_preg, errbuff, sizeof(errbuff)); + parser_errmsg("imfile: error in regex expansion: %s", errbuff); + ABORT_FINALIZE(RS_RET_ERR); + } + } if(inst->readTimeout != 0) loadModConf->haveReadTimeouts = 1; iRet = checkInstance(inst); @@ -1320,7 +1625,10 @@ CODESTARTbeginCnfLoad loadModConf->readTimeout = 0; /* default: no timeout */ loadModConf->timeoutGranularity = 1000; /* default: 1 second */ loadModConf->haveReadTimeouts = 0; /* default: no timeout */ + loadModConf->normalizePath = 1; loadModConf->sortFiles = GLOB_NOSORT; + loadModConf->conf_tree = calloc(sizeof(fs_node_t), 1); + loadModConf->conf_tree->edges = NULL; bLegacyCnfModGlobalsPermitted = 1; /* init legacy config vars */ cs.pszFileName = NULL; @@ -1370,6 +1678,8 @@ CODESTARTsetModCnf loadModConf->timeoutGranularity = (int) pvals[i].val.d.n * 1000; } else if(!strcmp(modpblk.descr[i].name, "sortfiles")) { loadModConf->sortFiles = ((sbool) pvals[i].val.d.n) ? 0 : GLOB_NOSORT; + } else if(!strcmp(modpblk.descr[i].name, "normalizepath")) { + loadModConf->normalizePath = (sbool) pvals[i].val.d.n; } else if(!strcmp(modpblk.descr[i].name, "mode")) { if(!es_strconstcmp(pvals[i].val.d.estr, "polling")) loadModConf->opMode = OPMODE_POLLING; @@ -1379,7 +1689,11 @@ CODESTARTsetModCnf DBGPRINTF("inotify mode configured, but only FEN " "is available on OS SOLARIS. Switching to FEN Mode automatically\n"); #else - loadModConf->opMode = OPMODE_INOTIFY; + #if defined(HAVE_INOTIFY_INIT) + loadModConf->opMode = OPMODE_INOTIFY; + #else + loadModConf->opMode = OPMODE_POLLING; + #endif #endif } else if(!es_strconstcmp(pvals[i].val.d.estr, "fen")) loadModConf->opMode = OPMODE_FEN; @@ -1447,19 +1761,31 @@ BEGINactivateCnf instanceConf_t *inst; CODESTARTactivateCnf runModConf = pModConf; - runModConf->pRootLstn = NULL, - runModConf->pTailLstn = NULL; + if(runModConf->root == NULL) { + LogError(0, NO_ERRCODE, "imfile: no file monitors configured, " + "input not activated.\n"); + ABORT_FINALIZE(RS_RET_NO_RUN); + } for(inst = runModConf->root ; inst != NULL ; inst = inst->next) { - addListner(inst); + // TODO: provide switch to turn off this warning? + if(!containsGlobWildcard((char*)inst->pszFileName)) { + if(access((char*)inst->pszFileName, R_OK) != 0) { + LogError(errno, RS_RET_ERR, + "imfile: on startup file '%s' does not exist " + "but is configured in static file monitor - this " + "may indicate a misconfiguration. If the file " + "appears at a later time, it will automatically " + "be processed. Reason", inst->pszFileName); + } + } + fs_node_add(runModConf->conf_tree, inst->pszFileName, 0, inst); } - /* if we could not set up any listeners, there is no point in running... */ - if(runModConf->pRootLstn == 0) { - LogError(0, NO_ERRCODE, "imfile: no file monitors could be started, " - "input not activated.\n"); - ABORT_FINALIZE(RS_RET_NO_RUN); + if(Debug) { + fs_node_print(runModConf->conf_tree, 0); } + finalize_it: ENDactivateCnf @@ -1467,14 +1793,19 @@ ENDactivateCnf BEGINfreeCnf instanceConf_t *inst, *del; CODESTARTfreeCnf + fs_node_destroy(pModConf->conf_tree); + //move_list_destruct(pModConf); for(inst = pModConf->root ; inst != NULL ; ) { + if(inst->startRegex != NULL) + regfree(&inst->end_preg); free(inst->pszBindRuleset); free(inst->pszFileName); - free(inst->pszDirName); - free(inst->pszFileBaseName); free(inst->pszTag); free(inst->pszStateFile); - free(inst->startRegex); + if(inst->startRegex != NULL) { + regfree(&inst->end_preg); + free(inst->startRegex); + } del = inst; inst = inst->next; free(del); @@ -1482,1108 +1813,150 @@ CODESTARTfreeCnf ENDfreeCnf -/* Monitor files in traditional polling mode. - * - * We go through all files and remember if at least one had data. If so, we do - * another run (until no data was present in any file). Then we sleep for - * PollInterval seconds and restart the whole process. This ensures that as - * long as there is some data present, it will be processed at the fastest - * possible pace - probably important for busy systmes. If we monitor just a - * single file, the algorithm is slightly modified. In that case, the sleep - * hapens immediately. The idea here is that if we have just one file, we - * returned from the file processer because that file had no additional data. - * So even if we found some lines, it is highly unlikely to find a new one - * just now. Trying it would result in a performance-costly additional try - * which in the very, very vast majority of cases will never find any new - * lines. - * On spamming the main queue: keep in mind that it will automatically rate-limit - * ourselfes if we begin to overrun it. So we really do not need to care here. - */ +/* Monitor files in polling mode. */ static rsRetVal doPolling(void) { - int bHadFileData; /* were there at least one file with data during this run? */ DEFiRet; while(glbl.GetGlobalInputTermState() == 0) { + DBGPRINTF("doPolling: new poll run\n"); do { - lstn_t *pLstn; - bHadFileData = 0; - for(pLstn = runModConf->pRootLstn ; pLstn != NULL ; pLstn = pLstn->next) { - if(glbl.GetGlobalInputTermState() == 1) - break; /* terminate input! */ - pollFile(pLstn, &bHadFileData); - } - } while(bHadFileData == 1 && glbl.GetGlobalInputTermState() == 0); - /* warning: do...while()! */ + runModConf->bHadFileData = 0; + fs_node_walk(runModConf->conf_tree, poll_tree); + DBGPRINTF("doPolling: end poll walk, hadData %d\n", runModConf->bHadFileData); + } while(runModConf->bHadFileData); /* warning: do...while()! */ /* Note: the additional 10ns wait is vitally important. It guards rsyslog * against totally hogging the CPU if the users selects a polling interval - * of 0 seconds. It doesn't hurt any other valid scenario. So do not remove. - * rgerhards, 2008-02-14 - */ - if(glbl.GetGlobalInputTermState() == 0) - srSleep(runModConf->iPollInterval, 10); - } - - RETiRet; -} - -//-------------------- NOW NEEDED for FEN as well! -#if defined(HAVE_INOTIFY_INIT) || (defined(OS_SOLARIS) && defined (HAVE_PORT_SOURCE_FILE)) -/* the basedir buffer must be of size MAXFNAME - * returns the index of the last slash before the basename - */ -static int ATTR_NONNULL() -getBasedir(uchar *const __restrict__ basedir, uchar *const __restrict__ path) -{ - int i; - int found = 0; - const int lenName = ustrlen(path); - for(i = lenName ; i >= 0 ; --i) { - if(path[i] == '/') { - /* found basename component */ - found = 1; - if(i == lenName) - basedir[0] = '\0'; - else { - memcpy(basedir, path, i); - basedir[i] = '\0'; - } - break; - } - } - if (found == 1) - return i; - else { - return -1; - } -} - -static rsRetVal ATTR_NONNULL() -fileTableInit(fileTable_t *const __restrict__ tab, const int nelem) -{ - DEFiRet; - CHKmalloc(tab->listeners = malloc(sizeof(dirInfoFiles_t) * nelem)); - tab->allocMax = nelem; - tab->currMax = 0; -finalize_it: - RETiRet; -} - -#if ULTRA_DEBUG == 1 -static void ATTR_NONNULL() -fileTableDisplay(fileTable_t *tab) -{ - int f; - uchar *baseName; - DBGPRINTF("fileTableDisplay = dirs.currMaxfiles %d\n", tab->currMax); - for(f = 0 ; f < tab->currMax ; ++f) { - baseName = tab->listeners[f].pLstn->pszBaseName; - DBGPRINTF("fileTableDisplay = TABLE %p CONTENTS, %d->%p:'%s'\n", - tab, f, tab->listeners[f].pLstn, (char*)baseName); - } -} -#endif - -static int ATTR_NONNULL() -fileTableSearch(fileTable_t *const __restrict__ tab, uchar *const __restrict__ fn) -{ - int f; - uchar *baseName = NULL; - #if ULTRA_DEBUG == 1 - fileTableDisplay(tab); - #endif - for(f = 0 ; f < tab->currMax ; ++f) { - baseName = tab->listeners[f].pLstn->pszBaseName; - if(!fnmatch((char*)baseName, (char*)fn, FNM_PATHNAME | FNM_PERIOD)) - break; /* found */ - } - if(f == tab->currMax) - f = -1; - DBGPRINTF("fileTableSearch file '%s' - '%s', found:%d\n", fn, (f == -1) ? NULL : baseName, f); - return f; -} - -static int ATTR_NONNULL() -fileTableSearchNoWildcard(fileTable_t *const __restrict__ tab, uchar *const __restrict__ fn) -{ - int f; - uchar *baseName = NULL; - for(f = 0 ; f < tab->currMax ; ++f) { - baseName = tab->listeners[f].pLstn->pszBaseName; - if (strcmp((const char*)baseName, (const char*)fn) == 0) - break; /* found */ - } - if(f == tab->currMax) - f = -1; - DBGPRINTF("fileTableSearchNoWildcard file '%s' - '%s', found:%d\n", fn, baseName, f); - return f; -} - -/* add file to file table */ -static rsRetVal ATTR_NONNULL() -fileTableAddFile(fileTable_t *const __restrict__ tab, lstn_t *const __restrict__ pLstn) -{ - int j; - DEFiRet; - for(j = 0 ; j < tab->currMax && tab->listeners[j].pLstn != pLstn ; ++j) - ; /* just scan */ - if(j < tab->currMax) { - ++tab->listeners[j].refcnt; - DBGPRINTF("file '%s' already registered, refcnt now %d\n", - pLstn->pszFileName, tab->listeners[j].refcnt); - FINALIZE; - } - - if(tab->currMax == tab->allocMax) { - const int newMax = 2 * tab->allocMax; - dirInfoFiles_t *newListenerTab = realloc(tab->listeners, newMax * sizeof(dirInfoFiles_t)); - if(newListenerTab == NULL) { - LogError(0, RS_RET_OUT_OF_MEMORY, - "cannot alloc memory to map directory/file relationship " - "for '%s' - ignoring", pLstn->pszFileName); - ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); - } - tab->listeners = newListenerTab; - tab->allocMax = newMax; - DBGPRINTF("increased dir table to %d entries\n", allocMaxDirs); - } - - tab->listeners[tab->currMax].pLstn = pLstn; - tab->listeners[tab->currMax].refcnt = 1; - tab->currMax++; -finalize_it: - RETiRet; -} - -/* delete a file from file table */ -static rsRetVal ATTR_NONNULL() -fileTableDelFile(fileTable_t *const __restrict__ tab, lstn_t *const __restrict__ pLstn) -{ - int j; - DEFiRet; - - for(j = 0 ; j < tab->currMax && tab->listeners[j].pLstn != pLstn ; ++j) - ; /* just scan */ - if(j == tab->currMax) { - DBGPRINTF("no association for file '%s'\n", pLstn->pszFileName); - FINALIZE; - } - tab->listeners[j].refcnt--; - if(tab->listeners[j].refcnt == 0) { - /* we remove that entry (but we never shrink the table) */ - if(j < tab->currMax - 1) { - /* entry in middle - need to move others */ - memmove(tab->listeners+j, tab->listeners+j+1, - (tab->currMax -j-1) * sizeof(dirInfoFiles_t)); - } - --tab->currMax; - } -finalize_it: - RETiRet; -} - -/* add entry to dirs array */ -static rsRetVal -dirsAdd(const uchar *const dirName, int *const piIndex) -{ - sbool sbAdded; - int newMax; - int i, newindex; - dirInfo_t *newDirTab; - char* psztmp; - DEFiRet; - - dbgprintf("dirsAdd: add '%s'\n", dirName); - /* Set new index to last dir by default, then search for a free spot in dirs array */ - newindex = currMaxDirs; - sbAdded = TRUE; - for(i= 0 ; i < currMaxDirs ; ++i) { - if (dirs[i].dirName == NULL) { - newindex = i; - sbAdded = FALSE; - DBGPRINTF("dirsAdd found free spot at %d, reusing\n", newindex); - break; - } - } - - /* Save Index for higher functions */ - if (piIndex != NULL ) - *piIndex = newindex; - - /* Check if dirstab size needs to be increased */ - if(sbAdded == TRUE && newindex == allocMaxDirs) { - newMax = 2 * allocMaxDirs; - newDirTab = realloc(dirs, newMax * sizeof(dirInfo_t)); - if(newDirTab == NULL) { - LogError(0, RS_RET_OUT_OF_MEMORY, - "dirsAdd cannot alloc memory to monitor directory '%s' - ignoring", - dirName); - ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); - } - dirs = newDirTab; - allocMaxDirs = newMax; - DBGPRINTF("dirsAdd increased dir table to %d entries\n", allocMaxDirs); - } - - /* if we reach this point, there is space in the file table for the new entry */ - CHKmalloc(dirs[newindex].dirName = ustrdup(dirName)); /* Get a copy of the string !*/ - dirs[newindex].dirNameBfWildCard = NULL; - dirs[newindex].bDirType = DIR_CONFIGURED; /* Default to configured! */ - CHKiRet(fileTableInit(&dirs[newindex].active, INIT_FILE_IN_DIR_TAB_SIZE)); - CHKiRet(fileTableInit(&dirs[newindex].configured, INIT_FILE_IN_DIR_TAB_SIZE)); - - /* check for wildcard in directoryname, if last character is a wildcard we remove it and try again! */ - dirs[newindex].hasWildcard = containsGlobWildcard((char*)dirName); - CHKmalloc(dirs[newindex].dirNameBfWildCard = ustrdup(dirName)); - - /* Default rootidx always -1 */ - dirs[newindex].rdiridx = -1; - - /* Extrac checking for wildcard mode */ - if (dirs[newindex].hasWildcard) { - // TODO: wildcard is not necessarily in last char!!! - // TODO: BUG: we have many more wildcards that "*" - so this check is invalid - DBGPRINTF("dirsAdd detected wildcard in dir '%s'\n", dirName); - - /* Set NULL Byte to FIRST wildcard occurrence */ - psztmp = strchr((char*)dirs[newindex].dirNameBfWildCard, '*'); - if (psztmp != NULL) { - *psztmp = '\0'; - /* Now set NULL Byte on last directory delimiter occurrence, - * This makes sure that we have the current base path to create a watch for! */ - psztmp = strrchr((char*)dirs[newindex].dirNameBfWildCard, '/'); - if (psztmp != NULL) { - *psztmp = '\0'; - } else { - LogError(0, RS_RET_SYS_ERR , "imfile: dirsAdd unexpected error #1 " - "for dir '%s' ", dirName); - ABORT_FINALIZE(RS_RET_SYS_ERR); - } - } else { - LogError(0, RS_RET_SYS_ERR , "imfile: dirsAdd unexpected error #2 " - "for dir '%s' ", dirName); - ABORT_FINALIZE(RS_RET_SYS_ERR); - } - DBGPRINTF("dirsAdd after wildcard removal: '%s'\n", - dirs[newindex].dirNameBfWildCard); - } - -#if defined(OS_SOLARIS) && defined (HAVE_PORT_SOURCE_FILE) - // Create FileInfo struct - dirs[newindex].pfinf = malloc(sizeof(struct fileinfo)); - if (dirs[newindex].pfinf == NULL) { - LogError(0, RS_RET_OUT_OF_MEMORY, "imfile: dirsAdd alloc memory " - "for fileinfo failed "); - ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); - } - /* Use parent dir on Wildcard pathes */ - if (dirs[newindex].hasWildcard) { - if ((dirs[newindex].pfinf->fobj.fo_name = strdup((char*)dirs[newindex].dirNameBfWildCard)) == NULL) { - LogError(0, RS_RET_OUT_OF_MEMORY, "imfile: dirsAdd alloc memory " - "for strdup failed "); - free(dirs[newindex].pfinf); - dirs[newindex].pfinf = NULL; - ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); - } - - } else { - if ((dirs[newindex].pfinf->fobj.fo_name = strdup((char*)dirs[newindex].dirName)) == NULL) { - LogError(0, RS_RET_OUT_OF_MEMORY, "imfile: dirsAdd alloc memory " - "for strdup failed "); - free(dirs[newindex].pfinf); - dirs[newindex].pfinf = NULL; - ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); - } - } - - /* Event types to watch. */ - dirs[newindex].pfinf->events = FILE_MODIFIED; // |FILE_DELETE|FILE_RENAME_TO|FILE_RENAME_FROM; - dirs[newindex].pfinf->port = glport; - dirs[newindex].bPortAssociated = 0; -#endif - - DBGPRINTF("dirsAdd added dir %d to dirs table: '%s'\n", newindex, dirName); - /* Increment only if entry was added and not reused */ - if (sbAdded == TRUE) - ++currMaxDirs; -finalize_it: - RETiRet; -} - - -/* checks if a dir name is already inside the dirs array. If so, returns - * its index. If not present, -1 is returned. - */ -static int ATTR_NONNULL(1) -dirsFindDir(const uchar *const dir) -{ - int i; - int iFind = -1; - - for(i = 0 ; i < currMaxDirs ; ++i) { - if (dirs[i].dirName != NULL && ustrcmp(dir, dirs[i].dirName) == 0) { - iFind = i; - break; - } - } - DBGPRINTF("dirsFindDir: '%s' - idx %d\n", dir, iFind); - return iFind; -} - -static rsRetVal -dirsInit(void) -{ - instanceConf_t *inst; - DEFiRet; - -DBGPRINTF("dirsInit\n"); - free(dirs); - CHKmalloc(dirs = malloc(sizeof(dirInfo_t) * INIT_FILE_TAB_SIZE)); - allocMaxDirs = INIT_FILE_TAB_SIZE; - currMaxDirs = 0; - - for(inst = runModConf->root ; inst != NULL ; inst = inst->next) { - if(dirsFindDir(inst->pszDirName) == -1) - dirsAdd(inst->pszDirName, NULL); - } - -finalize_it: - RETiRet; -} - -/* add file to directory (create association) - * fIdx is index into file table, all other information is pulled from that table. - * bActive is 1 if the file is to be added to active set, else zero - */ -static rsRetVal ATTR_NONNULL(1) -dirsAddFile(lstn_t *__restrict__ pLstn, const int bActive) -{ - int dirIdx; - dirInfo_t *dir; - DEFiRet; - - dirIdx = dirsFindDir(pLstn->pszDirName); - if(dirIdx == -1) { - LogError(0, RS_RET_INTERNAL_ERROR, "imfile: could not find " - "directory '%s' in dirs array - ignoring", - pLstn->pszDirName); - FINALIZE; - } - - dir = dirs + dirIdx; - CHKiRet(fileTableAddFile((bActive ? &dir->active : &dir->configured), pLstn)); - DBGPRINTF("associated file [%s] to directory %d[%s], Active = %d\n", - pLstn->pszFileName, dirIdx, dir->dirName, bActive); - #if ULTRA_DEBUG == 1 - /* UNCOMMENT FOR DEBUG fileTableDisplay(bActive ? &dir->active : &dir->configured); */ - #endif -finalize_it: - RETiRet; -} - -/* Duplicate an existing listener. This is called when a new file is to - * be monitored due to wildcard detection. Returns the new pLstn in - * the ppExisting parameter. - */ -static rsRetVal ATTR_NONNULL(1, 2) -lstnDup(lstn_t ** ppExisting, - const uchar *const __restrict__ newname, - const uchar *const __restrict__ newdirname) -{ - DEFiRet; - lstn_t *const existing = *ppExisting; - lstn_t *pThis; - CHKiRet(lstnAdd(&pThis)); - - /* Use dynamic dirname if newdirname is set! */ - if (newdirname == NULL) { - pThis->pszDirName = existing->pszDirName; /* read-only */ - } else { - CHKmalloc(pThis->pszDirName = ustrdup(newdirname)); - } - CHKmalloc(pThis->pszBaseName = ustrdup(newname)); - if(asprintf((char**)&pThis->pszFileName, "%s/%s", - (char*)pThis->pszDirName, (char*)newname) == -1) { - DBGPRINTF("lstnDup: asprintf failed, malfunction can happen\n"); - ABORT_FINALIZE(RS_RET_OUT_OF_MEMORY); - } - CHKmalloc(pThis->pszTag = ustrdup(existing->pszTag)); - pThis->lenTag = ustrlen(pThis->pszTag); - if(existing->pszStateFile == NULL) { - pThis->pszStateFile = NULL; - } else { - CHKmalloc(pThis->pszStateFile = ustrdup(existing->pszStateFile)); - } - - CHKiRet(ratelimitNew(&pThis->ratelimiter, "imfile", (char*)pThis->pszFileName)); - pThis->multiSub.maxElem = existing->multiSub.maxElem; - pThis->multiSub.nElem = 0; - CHKmalloc(pThis->multiSub.ppMsgs = MALLOC(pThis->multiSub.maxElem * sizeof(smsg_t*))); - pThis->iSeverity = existing->iSeverity; - pThis->iFacility = existing->iFacility; - pThis->maxLinesAtOnce = existing->maxLinesAtOnce; - pThis->trimLineOverBytes = existing->trimLineOverBytes; - pThis->iPersistStateInterval = existing->iPersistStateInterval; - pThis->readMode = existing->readMode; - pThis->startRegex = existing->startRegex; /* no strdup, as it is read-only */ - if(pThis->startRegex != NULL) // TODO: make this a single function with better error handling - if(regcomp(&pThis->end_preg, (char*)pThis->startRegex, REG_EXTENDED)) { - DBGPRINTF("error regex compile\n"); - ABORT_FINALIZE(RS_RET_ERR); - } - pThis->discardTruncatedMsg = existing->discardTruncatedMsg; - pThis->msgDiscardingError = existing->msgDiscardingError; - pThis->bRMStateOnDel = existing->bRMStateOnDel; - pThis->hasWildcard = existing->hasWildcard; - pThis->escapeLF = existing->escapeLF; - pThis->reopenOnTruncate = existing->reopenOnTruncate; - pThis->addMetadata = existing->addMetadata; - pThis->addCeeTag = existing->addCeeTag; - pThis->readTimeout = existing->readTimeout; - pThis->freshStartTail = existing->freshStartTail; - pThis->fileNotFoundError = existing->fileNotFoundError; - pThis->pRuleset = existing->pRuleset; - pThis->nRecords = 0; - pThis->pStrm = NULL; - pThis->prevLineSegment = NULL; - pThis->masterLstn = existing; - #ifdef HAVE_INOTIFY_INIT - pThis->movedfrom_statefile = NULL; - pThis->movedfrom_cookie = 0; - #endif - #if defined(OS_SOLARIS) && defined (HAVE_PORT_SOURCE_FILE) - pThis->pfinf = NULL; - pThis->bPortAssociated = 0; - #endif - *ppExisting = pThis; -finalize_it: - RETiRet; -} -#endif /* #if defined(HAVE_INOTIFY_INIT) || (defined(OS_SOLARIS) && defined (HAVE_PORT_SOURCE_FILE)) --- */ - -#ifdef HAVE_INOTIFY_INIT -static void -in_setupDirWatch(const int dirIdx) -{ - const int wd = inotify_add_watch(ino_fd, (char*)dirs[dirIdx].dirNameBfWildCard, - IN_CREATE|IN_DELETE|IN_MOVED_FROM|IN_MOVED_TO); - if(wd < 0) { - LogError(errno, RS_RET_IO_ERROR, "imfile: cannot watch directory '%s'", - dirs[dirIdx].dirNameBfWildCard); - goto done; - } - wdmapAdd(wd, dirIdx, NULL); - DBGPRINTF("in_setupDirWatch: watch %d added for dir %s(Idx=%d)\n", wd, - (char*) dirs[dirIdx].dirNameBfWildCard, dirIdx); -done: return; -} - -/* Setup a new file watch for a known active file. It must already have - * been entered into the correct tables. - * Note: we need to try to read this file, as it may already contain data this - * needs to be processed, and we won't get an event for that as notifications - * happen only for things after the watch has been activated. - * Note: newFileName is NULL for configured files, and non-NULL for dynamically - * detected files (e.g. wildcards!) - */ -static void ATTR_NONNULL(1) -startLstnFile(lstn_t *const __restrict__ pLstn) -{ - rsRetVal localRet; - DBGPRINTF("startLstnFile for file '%s'\n", pLstn->pszFileName); - const int wd = inotify_add_watch(ino_fd, (char*)pLstn->pszFileName, IN_MODIFY); - if(wd < 0) { - if(pLstn->fileNotFoundError) { - LogError(errno, NO_ERRCODE, "imfile: error with inotify API," - " ignoring file '%s'", pLstn->pszFileName); - } else { - char errStr[512]; - rs_strerror_r(errno, errStr, sizeof(errStr)); - DBGPRINTF("could not create file table entry for '%s' - " - "not processing it now: %s\n", pLstn->pszFileName, errStr); - } - goto done; - } - if((localRet = wdmapAdd(wd, -1, pLstn)) != RS_RET_OK) { - if( localRet != RS_RET_FILE_ALREADY_IN_TABLE && - pLstn->fileNotFoundError) { - LogError(0, NO_ERRCODE, "imfile: internal error: error %d adding file " - "to wdmap, ignoring file '%s'\n", localRet, pLstn->pszFileName); - } else { - DBGPRINTF("error %d adding file to wdmap, ignoring\n", localRet); - } - goto done; - } - DBGPRINTF("watch %d added for file %s\n", wd, pLstn->pszFileName); - dirsAddFile(pLstn, ACTIVE_FILE); - pollFile(pLstn, NULL); -done: return; -} - -/* Setup a new file watch for dynamically discovered files (via wildcards). - * Note: we need to try to read this file, as it may already contain data this - * needs to be processed, and we won't get an event for that as notifications - * happen only for things after the watch has been activated. - */ -static void ATTR_NONNULL(1) -in_setupFileWatchDynamic(lstn_t *pLstn, - uchar *const __restrict__ newBaseName, - uchar *const __restrict__ newFileName) -{ - char fullfn[MAXFNAME]; - struct stat fileInfo; - uchar basedir[MAXFNAME]; - uchar* pBaseDir = NULL; - int idirindex; - - if (newFileName == NULL) { - snprintf(fullfn, MAXFNAME, "%s/%s", pLstn->pszDirName, newBaseName); - } else { - /* Get BaseDir from filename! */ - getBasedir(basedir, newFileName); - pBaseDir = &basedir[0]; /* Set BaseDir Pointer */ - idirindex = dirsFindDir(basedir); - if (idirindex == -1) { - /* Add dir to table and create watch */ - DBGPRINTF("Adding new dir '%s' to dirs table \n", basedir); - dirsAdd(basedir, &idirindex); - in_setupDirWatch(idirindex); - } - /* Use newFileName */ - snprintf(fullfn, MAXFNAME, "%s", newFileName); - } - if(stat(fullfn, &fileInfo) != 0) { - char errStr[1024]; - rs_strerror_r(errno, errStr, sizeof(errStr)); - DBGPRINTF("ignoring file '%s' cannot stat(): %s\n", - fullfn, errStr); - goto done; - } - - if(S_ISDIR(fileInfo.st_mode)) { - DBGPRINTF("ignoring directory '%s'\n", fullfn); - goto done; - } - - if(lstnDup(&pLstn, newBaseName, pBaseDir) != RS_RET_OK) - goto done; - - startLstnFile(pLstn); -done: return; -} - -/* Search for matching files using glob. - * Create Dynamic Watch for each found file - */ -static void -in_setupFileWatchGlobSearch(lstn_t *pLstn) -{ - int wd; - - DBGPRINTF("in_setupFileWatchGlobSearch file '%s' has wildcard, doing initial expansion\n", - pLstn->pszFileName); - glob_t files; - const int ret = glob((char*)pLstn->pszFileName, - GLOB_MARK|GLOB_NOSORT|GLOB_BRACE, NULL, &files); - if(ret == 0) { - for(unsigned i = 0 ; i < files.gl_pathc ; i++) { - uchar basen[MAXFNAME]; - uchar *const file = (uchar*)files.gl_pathv[i]; - - if(file[strlen((char*)file)-1] == '/') - continue;/* we cannot process subdirs! */ - - /* search for existing watched files !*/ - wd = wdmapLookupFilename(file); - if(wd >= 0) { - DBGPRINTF("in_setupFileWatchGlobSearch '%s' already watched in wd %d\n", - file, wd); - } else { - getBasename(basen, file); - DBGPRINTF("in_setupFileWatchGlobSearch setup dynamic watch " - "for '%s : %s' \n", basen, file); - in_setupFileWatchDynamic(pLstn, basen, file); - } - } - globfree(&files); - } -} - -/* Setup a new file watch for static (configured) files. - * Note: we need to try to read this file, as it may already contain data this - * needs to be processed, and we won't get an event for that as notifications - * happen only for things after the watch has been activated. - */ -static void ATTR_NONNULL(1) -in_setupFileWatchStatic(lstn_t *pLstn) -{ - sbool hasWildcard; - - DBGPRINTF("in_setupFileWatchStatic: adding file '%s' to configured table\n", - pLstn->pszFileName); - dirsAddFile(pLstn, CONFIGURED_FILE); - - /* perform wildcard check on static files manually */ - hasWildcard = containsGlobWildcard((char*)pLstn->pszFileName); - if(hasWildcard) { - /* search for matching files */ - in_setupFileWatchGlobSearch(pLstn); - } else { - /* Duplicate static object as well, otherwise the configobject - * could be deleted later! */ - if(lstnDup(&pLstn, pLstn->pszBaseName, NULL) != RS_RET_OK) { - DBGPRINTF("in_setupFileWatchStatic failed to duplicate listener for '%s'\n", - pLstn->pszFileName); - goto done; - } - startLstnFile(pLstn); - } -done: return; -} - -/* setup our initial set of watches, based on user config */ -static void -in_setupInitialWatches(void) -{ - int i; - DBGPRINTF("setting up initial directory watches\n"); - for(i = 0 ; i < currMaxDirs ; ++i) { - in_setupDirWatch(i); - } - lstn_t *pLstn; - DBGPRINTF("setting up initial listener watches\n"); - for(pLstn = runModConf->pRootLstn ; pLstn != NULL ; pLstn = pLstn->next) { - if(pLstn->masterLstn == NULL) { - /* we process only static (master) entries */ - in_setupFileWatchStatic(pLstn); - } + * of 0 seconds. It doesn't hurt any other valid scenario. So do not remove. + * rgerhards, 2008-02-14 + */ + DBGPRINTF("doPolling: poll going to sleep\n"); + if(glbl.GetGlobalInputTermState() == 0) + srSleep(runModConf->iPollInterval, 10); } + + RETiRet; } +#if defined(HAVE_INOTIFY_INIT) + static void ATTR_NONNULL(1) -in_dbg_showEv(struct inotify_event *ev) +in_dbg_showEv(const struct inotify_event *ev) { if(!Debug) return; if(ev->mask & IN_IGNORED) { dbgprintf("INOTIFY event: watch was REMOVED\n"); - } else if(ev->mask & IN_MODIFY) { + } + if(ev->mask & IN_MODIFY) { dbgprintf("INOTIFY event: watch was MODIFID\n"); - } else if(ev->mask & IN_ACCESS) { + } + if(ev->mask & IN_ACCESS) { dbgprintf("INOTIFY event: watch IN_ACCESS\n"); - } else if(ev->mask & IN_ATTRIB) { + } + if(ev->mask & IN_ATTRIB) { dbgprintf("INOTIFY event: watch IN_ATTRIB\n"); - } else if(ev->mask & IN_CLOSE_WRITE) { + } + if(ev->mask & IN_CLOSE_WRITE) { dbgprintf("INOTIFY event: watch IN_CLOSE_WRITE\n"); - } else if(ev->mask & IN_CLOSE_NOWRITE) { + } + if(ev->mask & IN_CLOSE_NOWRITE) { dbgprintf("INOTIFY event: watch IN_CLOSE_NOWRITE\n"); - } else if(ev->mask & IN_CREATE) { + } + if(ev->mask & IN_CREATE) { dbgprintf("INOTIFY event: file was CREATED: %s\n", ev->name); - } else if(ev->mask & IN_DELETE) { + } + if(ev->mask & IN_DELETE) { dbgprintf("INOTIFY event: watch IN_DELETE\n"); - } else if(ev->mask & IN_DELETE_SELF) { + } + if(ev->mask & IN_DELETE_SELF) { dbgprintf("INOTIFY event: watch IN_DELETE_SELF\n"); - } else if(ev->mask & IN_MOVE_SELF) { + } + if(ev->mask & IN_MOVE_SELF) { dbgprintf("INOTIFY event: watch IN_MOVE_SELF\n"); - } else if(ev->mask & IN_MOVED_FROM) { - dbgprintf("INOTIFY event: watch IN_MOVED_FROM\n"); - } else if(ev->mask & IN_MOVED_TO) { - dbgprintf("INOTIFY event: watch IN_MOVED_TO\n"); - } else if(ev->mask & IN_OPEN) { + } + if(ev->mask & IN_MOVED_FROM) { + dbgprintf("INOTIFY event: watch IN_MOVED_FROM, cookie %u, name '%s'\n", ev->cookie, ev->name); + } + if(ev->mask & IN_MOVED_TO) { + dbgprintf("INOTIFY event: watch IN_MOVED_TO, cookie %u, name '%s'\n", ev->cookie, ev->name); + } + if(ev->mask & IN_OPEN) { dbgprintf("INOTIFY event: watch IN_OPEN\n"); - } else if(ev->mask & IN_ISDIR) { + } + if(ev->mask & IN_ISDIR) { dbgprintf("INOTIFY event: watch IN_ISDIR\n"); - } else { - dbgprintf("INOTIFY event: unknown mask code %8.8x\n", ev->mask); - } -} - -/* Helper function to get fullpath when handling inotify dir events */ -static void ATTR_NONNULL() -in_handleDirGetFullDir(char *const pszoutput, const int dirIdx, const char *const pszsubdir) -{ - assert(dirIdx >= 0); - DBGPRINTF("in_handleDirGetFullDir root='%s' sub='%s' \n", dirs[dirIdx].dirName, pszsubdir); - snprintf(pszoutput, MAXFNAME, "%s/%s", dirs[dirIdx].dirNameBfWildCard, pszsubdir); -} - -/* inotify told us that a file's wd was closed. We now need to remove - * the file from our internal structures. Remember that a different inode - * with the same name may already be in processing. - */ -static void ATTR_NONNULL(2) -in_removeFile(const int dirIdx, lstn_t *const __restrict__ pLstn, const sbool bRemoveStateFile) -{ - uchar statefile[MAXFNAME]; - uchar toDel[MAXFNAME]; - int bDoRMState; - int wd; - uchar *statefn; - - DBGPRINTF("remove listener '%s', dirIdx %d\n", pLstn->pszFileName, dirIdx); - if(bRemoveStateFile == TRUE && pLstn->bRMStateOnDel) { - statefn = getStateFileName(pLstn, statefile, sizeof(statefile), NULL); - /* Get full path and file name */ - getFullStateFileName(statefn, toDel, sizeof(toDel)); - bDoRMState = 1; - } else { - bDoRMState = 0; - } - pollFile(pLstn, NULL); /* one final try to gather data */ - /* delete listener data */ - DBGPRINTF("DELETING listener data for '%s' - '%s'\n", pLstn->pszBaseName, pLstn->pszFileName); - lstnDel(pLstn); - fileTableDelFile(&dirs[dirIdx].active, pLstn); - if(bDoRMState) { - DBGPRINTF("unlinking '%s'\n", toDel); - if(unlink((char*)toDel) != 0) { - LogError(errno, RS_RET_ERR, "imfile: could not remove state " - "file '%s'", toDel); - } } - - wd = wdmapLookupListner(pLstn); - wdmapDel(wd); } -static void ATTR_NONNULL(1) -in_handleDirEventDirCREATE(struct inotify_event *ev, const int dirIdx) -{ - char fulldn[MAXFNAME]; - int newdiridx; - int iListIdx; - sbool hasWildcard; - - /* Combine to Full Path first */ - in_handleDirGetFullDir(fulldn, dirIdx, (char*)ev->name); - - /* Search for existing entry first! */ - newdiridx = dirsFindDir( (uchar*)fulldn ); - if(newdiridx == -1) { - /* Add dir to table and create watch */ - DBGPRINTF("in_handleDirEventDirCREATE Adding new dir '%s' to dirs table \n", fulldn); - dirsAdd((uchar*)fulldn, &newdiridx); - dirs[newdiridx].bDirType = DIR_DYNAMIC; /* Set to DYNAMIC directory! */ - in_setupDirWatch(newdiridx); - /* Set propper root index for newdiridx */ - dirs[newdiridx].rdiridx = (dirs[dirIdx].rdiridx != -1 ? dirs[dirIdx].rdiridx : dirIdx); - - DBGPRINTF("in_handleDirEventDirCREATE wdentry dirIdx=%d, rdirIdx=%d, dirIdxName=%s, dir=%s)\n", - dirIdx, dirs[newdiridx].rdiridx, dirs[dirIdx].dirName, fulldn); - if (dirs[dirs[newdiridx].rdiridx].configured.currMax > 0) { - DBGPRINTF("in_handleDirEventDirCREATE found configured listeners\n"); - - /* Loop through configured Listeners and scan for dynamic files */ - for(iListIdx = 0; iListIdx < dirs[dirs[newdiridx].rdiridx].configured.currMax; iListIdx++) { - hasWildcard = ( dirs[dirs[newdiridx].rdiridx].hasWildcard || - dirs[dirs[newdiridx].rdiridx].configured.listeners[iListIdx].pLstn->hasWildcard - ? TRUE : FALSE); - if (hasWildcard == 1){ - DBGPRINTF("in_handleDirEventDirCREATE configured listener has Wildcard\n"); - /* search for matching files */ - in_setupFileWatchGlobSearch( - dirs[dirs[newdiridx].rdiridx].configured.listeners[iListIdx].pLstn); - } - } - } - } else { - DBGPRINTF("dir '%s' already exists in dirs table (Idx %d)\n", fulldn, newdiridx); - } -} -static void ATTR_NONNULL(1) -in_handleDirEventFileCREATE(struct inotify_event *const ev, const int dirIdx) +static void ATTR_NONNULL(1, 2) +in_handleFileEvent(struct inotify_event *ev, const wd_map_t *const etry) { - int i; - lstn_t *pLstn = NULL; - int ftIdx; - char fullfn[MAXFNAME]; - uchar statefile_new[MAXFNAME]; - uchar statefilefull_old[MAXFNAME]; - uchar statefilefull_new[MAXFNAME]; - uchar* pszDir = NULL; - int dirIdxFinal = dirIdx; - ftIdx = fileTableSearch(&dirs[dirIdxFinal].active, (uchar*)ev->name); - if(ftIdx >= 0) { - pLstn = dirs[dirIdxFinal].active.listeners[ftIdx].pLstn; + if(ev->mask & IN_MODIFY) { + DBGPRINTF("fs_node_notify_file_update: act->name '%s'\n", etry->act->name); + pollFile(etry->act); } else { - DBGPRINTF("in_handleDirEventFileCREATE '%s' not associated with dir '%s' " - "(CurMax:%d, DirIdx:%d, DirType:%s)\n", ev->name, dirs[dirIdxFinal].dirName, - dirs[dirIdxFinal].active.currMax, dirIdxFinal, - (dirs[dirIdxFinal].bDirType == DIR_CONFIGURED ? "configured" : "dynamic") ); - ftIdx = fileTableSearch(&dirs[dirIdxFinal].configured, (uchar*)ev->name); - if(ftIdx == -1) { - if (dirs[dirIdxFinal].bDirType == DIR_DYNAMIC) { - /* Search all other configured directories for proper index! */ - if (currMaxDirs > 0) { - /* Store Dirname as we need to overwrite it in in_setupFileWatchDynamic */ - pszDir = dirs[dirIdxFinal].dirName; - - /* Combine directory and filename */ - snprintf(fullfn, MAXFNAME, "%s/%s", pszDir, (uchar*)ev->name); - - for(i = 0 ; i < currMaxDirs ; ++i) { - ftIdx = fileTableSearch(&dirs[i].configured, (uchar*)ev->name); - if(ftIdx != -1) { - /* Found matching directory! */ - dirIdxFinal = i; /* Have to correct directory index for - listnr dupl in in_setupFileWatchDynamic */ - - DBGPRINTF("Found matching directory for file '%s' in " - "dir '%s' (Idx=%d)\n", ev->name, - dirs[dirIdxFinal].dirName, dirIdxFinal); - break; - } - } - /* Found Listener to se */ - pLstn = dirs[dirIdxFinal].configured.listeners[ftIdx].pLstn; - - if(ftIdx == -1) { - DBGPRINTF("file '%s' not associated with dir '%s' and also no " - "matching wildcard directory found\n", ev->name, - dirs[dirIdxFinal].dirName); - goto done; - } - else { - DBGPRINTF("file '%s' not associated with dir '%s', using " - "dirIndex %d instead\n", ev->name, (pszDir == NULL) - ? dirs[dirIdxFinal].dirName : pszDir, dirIdxFinal); - } - } else { - DBGPRINTF("file '%s' not associated with dir '%s'\n", - ev->name, dirs[dirIdxFinal].dirName); - goto done; - } - } - } else - pLstn = dirs[dirIdxFinal].configured.listeners[ftIdx].pLstn; - } - if (pLstn != NULL) { - DBGPRINTF("file '%s' associated with dir '%s' (Idx=%d)\n", - ev->name, (pszDir == NULL) ? dirs[dirIdxFinal].dirName : pszDir, dirIdxFinal); - - /* We need to check if we have a preexisting statefile and move it*/ - if(ev->mask & IN_MOVED_TO) { - if (pLstn->movedfrom_statefile != NULL && pLstn->movedfrom_cookie == ev->cookie) { - /* We need to prepar fullfn before we can generate statefilename */ - snprintf(fullfn, MAXFNAME, "%s/%s", (pszDir == NULL) ? dirs[dirIdxFinal].dirName - : pszDir, (uchar*)ev->name); - getStateFileName(NULL, statefile_new, sizeof(statefile_new), (uchar*)fullfn); - getFullStateFileName(statefile_new, statefilefull_new, sizeof(statefilefull_new)); - getFullStateFileName(pLstn->movedfrom_statefile, statefilefull_old, - sizeof(statefilefull_old)); - - DBGPRINTF("old statefile '%s' needs to be moved to '%s' first!\n", - statefilefull_old, statefilefull_new); - -// openStateFileAndMigrate(pLstn, &statefilefull_old[0], &statefilefull_new[0]); - if(rename((char*) &statefilefull_old, (char*) &statefilefull_new) != 0) { - LogError(errno, RS_RET_ERR, "imfile: could not rename statefile " - "'%s' into '%s'", statefilefull_old, statefilefull_new); - } else { - DBGPRINTF("statefile '%s' renamed into '%s'\n", statefilefull_old, - statefilefull_new); - } - - /* Free statefile memory */ - free(pLstn->movedfrom_statefile); - pLstn->movedfrom_statefile = NULL; - pLstn->movedfrom_cookie = 0; - } else { - DBGPRINTF("IN_MOVED_TO either unknown cookie '%d' we expected '%d' or " - "missing statefile '%s'\n", pLstn->movedfrom_cookie, - ev->cookie, pLstn->movedfrom_statefile); - } - } - - /* Setup a watch now for new file*/ - in_setupFileWatchDynamic(pLstn, (uchar*)ev->name, (pszDir == NULL) ? NULL : (uchar*)fullfn); + DBGPRINTF("got non-expected inotify event:\n"); + in_dbg_showEv(ev); } -done: return; } -/* note: we need to care only for active files in the DELETE case. - * Two reasons: a) if this is a configured file, it should be active - * b) if not for some reason, there still is nothing we can do against - * it, and trying to process a *deleted* file really makes no sense - * (remeber we don't have it open, so it actually *is gone*). - */ -static void ATTR_NONNULL(1) -in_handleDirEventFileDELETE(struct inotify_event *const ev, const int dirIdx) -{ - const int ftIdx = fileTableSearch(&dirs[dirIdx].active, (uchar*)ev->name); - if(ftIdx == -1) { - DBGPRINTF("deleted file '%s' not active in dir '%s'\n", - ev->name, dirs[dirIdx].dirName); - goto done; - } - DBGPRINTF("imfile delete processing for '%s'\n", - dirs[dirIdx].active.listeners[ftIdx].pLstn->pszFileName); - in_removeFile(dirIdx, dirs[dirIdx].active.listeners[ftIdx].pLstn, TRUE); -done: return; -} -/* inotify told us that a dirs's wd was closed. We now need to remove - * the dir from our internal structures. +/* workaround for IN_MOVED: walk active list and prevent state file deletion of + * IN_MOVED_IN active object + * TODO: replace by a more generic solution. */ static void -in_removeDir(const int dirIdx) -{ - int wd; - wd = wdmapLookupListnerIdx(dirIdx); - DBGPRINTF("in_removeDir remove '%s', dirIdx=%d wdindex=%d\n", dirs[dirIdx].dirName, dirIdx, wd); - wdmapDel(wd); -} - -static void ATTR_NONNULL(1) -in_handleDirEventDirDELETE(struct inotify_event *const ev, const int dirIdx) +flag_in_move(fs_edge_t *const edge, const char *name_moved) { - char fulldn[MAXFNAME]; - int finddiridx; - - in_handleDirGetFullDir(fulldn, dirIdx, (char*)ev->name); + act_obj_t *act; - /* Search for existing entry first! */ - finddiridx = dirsFindDir( (uchar*)fulldn ); - - if(finddiridx != -1) { - /* Remove internal structures */ - in_removeDir(finddiridx); - - /* Delete dir from dirs array! */ - free(dirs[finddiridx].dirName); - if (dirs[finddiridx].dirNameBfWildCard != NULL) - free(dirs[finddiridx].dirNameBfWildCard); - free(dirs[finddiridx].active.listeners); - free(dirs[finddiridx].configured.listeners); - dirs[finddiridx].dirName = NULL; - dirs[finddiridx].dirNameBfWildCard = NULL; - - DBGPRINTF("in_handleDirEventDirDELETE dir (idx %d) '%s' deleted \n", finddiridx, fulldn); - } else { - DBGPRINTF("in_handleDirEventDirDELETE ERROR could not found '%s' in dirs table!\n", fulldn); - } -} - -static void ATTR_NONNULL(1) -in_handleDirEvent(struct inotify_event *const ev, const int dirIdx) -{ - DBGPRINTF("in_handleDirEvent dir event for (Idx %d)%s (mask %x)\n", - dirIdx, dirs[dirIdx].dirName, ev->mask); - if((ev->mask & IN_CREATE)) { - /* TODO: does IN_MOVED_TO make sense here ? */ - if((ev->mask & IN_ISDIR) || (ev->mask & IN_MOVED_TO)) { - in_handleDirEventDirCREATE(ev, dirIdx); /* Create new Dir */ - } else { - in_handleDirEventFileCREATE(ev, dirIdx); /* Create new File */ - } - } else if((ev->mask & IN_DELETE)) { - if((ev->mask & IN_ISDIR)) { - in_handleDirEventDirDELETE(ev, dirIdx); /* Create new Dir */ - } else { - in_handleDirEventFileDELETE(ev, dirIdx);/* Delete File from dir filetable */ - } - } else if((ev->mask & IN_MOVED_TO)) { - if((ev->mask & IN_ISDIR)) { - in_handleDirEventDirCREATE(ev, dirIdx); /* Create new Dir */ + for(act = edge->active ; act != NULL ; act = act->next) { + DBGPRINTF("checking active object %s\n", act->basename); + if(!strcmp(act->basename, name_moved)){ + DBGPRINTF("found file\n"); + act->in_move = 1; + break; } else { - in_handleDirEventFileCREATE(ev, dirIdx); /* Create new File */ + DBGPRINTF("name check fails, '%s' != '%s'\n", act->basename, name_moved); } - } else { - DBGPRINTF("got non-expected inotify event:\n"); - in_dbg_showEv(ev); - } -} - - -static void ATTR_NONNULL(1, 2) -in_handleFileEvent(struct inotify_event *ev, const wd_map_t *const etry) -{ - if(ev->mask & IN_MODIFY) { - pollFile(etry->pLstn, NULL); - } else { - DBGPRINTF("got non-expected inotify event:\n"); - in_dbg_showEv(ev); } } static void ATTR_NONNULL(1) in_processEvent(struct inotify_event *ev) { - wd_map_t *etry; - lstn_t *pLstn; - int iRet; - int ftIdx; - int wd; - uchar statefile[MAXFNAME]; - if(ev->mask & IN_IGNORED) { - goto done; - } else if(ev->mask & IN_MOVED_FROM) { - /* Find wd entry and remove it */ - etry = wdmapLookup(ev->wd); - if(etry != NULL) { - ftIdx = fileTableSearchNoWildcard(&dirs[etry->dirIdx].active, (uchar*)ev->name); - DBGPRINTF("IN_MOVED_FROM Event (ftIdx=%d, name=%s)\n", ftIdx, ev->name); - if(ftIdx >= 0) { - /* Find listener and wd table index*/ - pLstn = dirs[etry->dirIdx].active.listeners[ftIdx].pLstn; - wd = wdmapLookupListner(pLstn); - - /* Remove file from inotify watch */ - iRet = inotify_rm_watch(ino_fd, wd); /* Note this will TRIGGER IN_IGNORED Event! */ - if (iRet != 0) { - DBGPRINTF("inotify_rm_watch error %d (ftIdx=%d, wd=%d, name=%s)\n", - errno, ftIdx, wd, ev->name); - } else { - DBGPRINTF("inotify_rm_watch successfully removed file from watch " - "(ftIdx=%d, wd=%d, name=%s)\n", ftIdx, wd, ev->name); - } - - /* Store statefile name for later MOVED_TO event along with COOKIE */ - pLstn->masterLstn->movedfrom_statefile = ustrdup(getStateFileName(pLstn, - statefile, sizeof(statefile), NULL) ); - pLstn->masterLstn->movedfrom_cookie = ev->cookie; - - /* do NOT remove statefile in this case!*/ - in_removeFile(etry->dirIdx, pLstn, FALSE); - DBGPRINTF("IN_MOVED_FROM Event file removed file (wd=%d, name=%s)\n", wd, ev->name); - } - } + DBGPRINTF("imfile: got IN_IGNORED event\n"); goto done; } - DBGPRINTF("in_processEvent process Event %x for %s\n", ev->mask, (uchar*)ev->name); - etry = wdmapLookup(ev->wd); + DBGPRINTF("in_processEvent process Event %x for %s\n", ev->mask, ev->name); + const wd_map_t *const etry = wdmapLookup(ev->wd); if(etry == NULL) { - DBGPRINTF("could not lookup wd %d\n", ev->wd); + LogMsg(0, RS_RET_INTERNAL_ERROR, LOG_WARNING, "imfile: internal error? " + "inotify provided watch descriptor %d which we could not find " + "in our tables - ignored", ev->wd); goto done; } - if(etry->pLstn == NULL) { /* directory? */ - in_handleDirEvent(ev, etry->dirIdx); + DBGPRINTF("in_processEvent process Event %x is_file %d, act->name '%s'\n", + ev->mask, etry->act->edge->is_file, etry->act->name); + + if((ev->mask & IN_MOVED_FROM)) { + flag_in_move(etry->act->edge->node->edges, ev->name); + } + if(ev->mask & (IN_MOVED_FROM | IN_MOVED_TO)) { + fs_node_walk(etry->act->edge->node, poll_tree); + } else if(etry->act->edge->is_file) { + in_handleFileEvent(ev, etry); // esentially poll_file()! } else { - in_handleFileEvent(ev, etry); + fs_node_walk(etry->act->edge->node, poll_tree); } done: return; } -static void -in_do_timeout_processing(void) -{ - int i; - DBGPRINTF("readTimeouts are configured, checking if some apply\n"); - - for(i = 0 ; i < nWdmap ; ++i) { - DBGPRINTF("wdmap %d, plstn %p\n", i, wdmap[i].pLstn); - lstn_t *const pLstn = wdmap[i].pLstn; - if(pLstn != NULL && strmReadMultiLine_isTimedOut(pLstn->pStrm)) { - DBGPRINTF("wdmap %d, timeout occured\n", i); - pollFile(pLstn, NULL); - } - } -} - /* Monitor files in inotify mode */ #if !defined(_AIX) @@ -2600,11 +1973,7 @@ do_inotify(void) int currev; DEFiRet; -dbgprintf("pre wdmapinit\n"); CHKiRet(wdmapInit()); -dbgprintf("pre dirsinit\n"); - CHKiRet(dirsInit()); -dbgprintf("pre inotify_init\n"); ino_fd = inotify_init(); if(ino_fd < 0) { LogError(errno, RS_RET_INOTIFY_INIT_FAILED, "imfile: Init inotify " @@ -2612,9 +1981,9 @@ dbgprintf("pre inotify_init\n"); return RS_RET_INOTIFY_INIT_FAILED; } DBGPRINTF("inotify fd %d\n", ino_fd); -dbgprintf("pre setuInitialWatches\n"); - in_setupInitialWatches(); -dbgprintf("post setuInitialWatches\n"); + + /* do watch initialization */ + fs_node_walk(runModConf->conf_tree, poll_tree); while(glbl.GetGlobalInputTermState() == 0) { if(runModConf->haveReadTimeouts) { @@ -2626,7 +1995,8 @@ dbgprintf("post setuInitialWatches\n"); r = poll(&pollfd, 1, runModConf->timeoutGranularity); } while(r == -1 && errno == EINTR); if(r == 0) { - in_do_timeout_processing(); + DBGPRINTF("readTimeouts are configured, checking if some apply\n"); + fs_node_walk(runModConf->conf_tree, poll_timeouts); continue; } else if (r == -1) { LogError(errno, RS_RET_INTERNAL_ERROR, @@ -2709,499 +2079,71 @@ fen_printevent(int event) } } -static rsRetVal ATTR_NONNULL(1) -fen_removeFile(lstn_t *pLstn) -{ - struct stat statFile; - uchar statefile[MAXFNAME]; - uchar toDel[MAXFNAME]; - int bDoRMState; - uchar *statefn; - int dirIdx; - int ftIdx; - DEFiRet; - - /* Get correct dirindex from basedir! */ - dirIdx = dirsFindDir(pLstn->pszDirName); - if (dirIdx == -1) { - /* Add error processing as required, file may have been deleted/moved. */ - LogError(errno, RS_RET_SYS_ERR, "fen_removeFile: Failed to remove file, " - "unknown directory index for dir '%s'\n", pLstn->pszDirName); - ABORT_FINALIZE(RS_RET_SYS_ERR); - } - - if(pLstn->bRMStateOnDel) { - statefn = getStateFileName(pLstn, statefile, sizeof(statefile), NULL); - /* Get full path and file name */ - getFullStateFileName(statefn, toDel, sizeof(toDel)); - bDoRMState = 1; - } else { - bDoRMState = 0; - } - - /* one final try to gather data */ - pollFile(pLstn, NULL); - - /* Check for active / configure file */ - if ( (ftIdx = fileTableSearchNoWildcard(&dirs[dirIdx].active, pLstn->pszBaseName)) >= 0) { - DBGPRINTF("fen_removeFile removing active listener for '%s', dirIdx %d, ftIdx %d\n", - pLstn->pszFileName, dirIdx, ftIdx); - fileTableDelFile(&dirs[dirIdx].active, pLstn); /* Remove from filetable */ - lstnDel(pLstn); /* Delete Listener now */ - } else { - DBGPRINTF("fen_removeFile NOT removing configured listener for '%s', dirIdx %d\n", - pLstn->pszFileName, dirIdx, ftIdx); - } - - if(bDoRMState) { - DBGPRINTF("fen_removeFile unlinking '%s'\n", toDel); - if((stat((const char*) toDel, &statFile) == 0) && unlink((char*)toDel) != 0) { - LogError(errno, RS_RET_ERR, "fen_removeFile: could not remove state " - "file \"%s\": %s", toDel); - } - } -finalize_it: - RETiRet; -} - -static rsRetVal -fen_removeDir(int dirIdx) -{ - DEFiRet; - - DBGPRINTF("fen_removeDir removing dir (idx %d) '%s' \n", dirIdx, dirs[dirIdx].dirName); - - /* Delete dir from dirs array! */ - free(dirs[dirIdx].dirName); - free(dirs[dirIdx].dirNameBfWildCard); - free(dirs[dirIdx].active.listeners); - free(dirs[dirIdx].configured.listeners); - dirs[dirIdx].dirName = NULL; - dirs[dirIdx].dirNameBfWildCard = NULL; -finalize_it: - RETiRet; -} - -static rsRetVal -fen_processEventFile(struct file_obj* fobjp, lstn_t *pLstn, int revents, int dirIdx) -{ - struct stat statFile; - int ftIdx; - DEFiRet; - - // Use FileObj from listener if NULL - if (fobjp== NULL) - fobjp = &pLstn->pfinf->fobj; - - /* uncomment if needed DBGPRINTF("fen_processEventFile: %s (0x%" PRIXPTR ") ", fobjp->fo_name, pLstn); **/ - DBGPRINTF("fen_processEventFile: %s ", fobjp->fo_name); - if (revents) { - fen_printevent(revents); - } - DBGPRINTF("\n"); - - if (pLstn == NULL) { - /* Should not be NULL but it case it is abort */ - DBGPRINTF("fen_processEventFile: Listener '%s' for EVENT already deleted, aborting function.\n", - fobjp->fo_name); - FINALIZE; - } - - /* Port needs to be reassociated */ - pLstn->bPortAssociated = 0; - - /* Compare filename first */ - if (strcmp(fobjp->fo_name, (const char*)pLstn->pszFileName) == 0){ - DBGPRINTF("fen_processEventFile: matching file found: '%s'\n", fobjp->fo_name); - - /* Get File Stats */ - if (!(revents & FILE_EXCEPTION) && stat(fobjp->fo_name, &statFile) == -1) { - const int errno_save = errno; - DBGPRINTF("fen_processEventFile: Failed to stat file: %s - errno %d\n", - fobjp->fo_name, errno); - LogError(errno_save, RS_RET_FILE_NO_STAT, "imfile: file '%s' not found when " - "receiving notification event", fobjp->fo_name); - ABORT_FINALIZE(RS_RET_FILE_NO_STAT); - } - - /* - * Add what ever processing that needs to be done - * here. Process received events. - */ - if (revents) { - if (revents & FILE_MODIFIED) { - /* File has been modified, trigger a pollFile */ - pollFile(pLstn, NULL); - } else if (revents & FILE_RENAME_FROM) { - /* File has been renamed which means it was deleted. remove the file*/ - fen_removeFile(pLstn); - FINALIZE; - } - - /* check for file exception. Only happens when really bad things happened like - harddisk removal. We need to re-register he port. */ - if (revents & FILE_EXCEPTION) { - fen_removeFile(pLstn); - ABORT_FINALIZE(RS_RET_SYS_ERR); - } - } - } else { - DBGPRINTF("fen_processEventFile: file '%s' did not match Listener Filename '%s'\n", - fobjp->fo_name, pLstn->pszFileName); - ABORT_FINALIZE(RS_RET_SYS_ERR); - } - - /* Register file event */ - fobjp->fo_atime = statFile.st_atim; - fobjp->fo_mtime = statFile.st_mtim; - fobjp->fo_ctime = statFile.st_ctim; - if (port_associate(glport, PORT_SOURCE_FILE, (uintptr_t)fobjp, - pLstn->pfinf->events, (void *)pLstn) == -1) { - /* Add error processing as required, file may have been deleted/moved. */ - LogError(errno, RS_RET_SYS_ERR, "fen_processEventFile: Failed to associated port for file " - ": %s - errno %d\n", fobjp->fo_name, errno); - ABORT_FINALIZE(RS_RET_SYS_ERR); - } else { - /* Port successfull listening now*/ - DBGPRINTF("fen_processEventFile: associated port for file %s\n", fobjp->fo_name); - pLstn->bPortAssociated = 1; - } -finalize_it: - RETiRet; -} - -/* Helper function to find matching files for listener */ -rsRetVal -fen_DirSearchFiles(lstn_t *pLstn, int dirIdx) -{ - struct file_obj *fobjp = NULL; /* Helper object */ - rsRetVal localRet; - int ftIdx; - int result; - glob_t files; - lstn_t *pLstnNew; - /* Helper chars */ - uchar basedir[MAXFNAME]; - uchar basefilename[MAXFNAME]; - uchar *file; - DEFiRet; - - DBGPRINTF("fen_DirSearchFiles search for dynamic files with pattern '%s' \n", pLstn->pszFileName); - result = glob( (char*)pLstn->pszFileName, - GLOB_MARK|runModConf->sortFiles|GLOB_BRACE, - NULL, &files); - if(result == 0) { - DBGPRINTF("fen_DirSearchFiles found %d matches with pattern '%s' \n", files.gl_pathc, - pLstn->pszFileName); - for(unsigned i = 0 ; i < files.gl_pathc ; i++) { - /* Get File, Dir and Basename */ - file = (uchar*)files.gl_pathv[i]; - getBasename(basefilename, file); - getBasedir(basedir, file); - - if(file[strlen((char*)file)-1] == '/') - continue;/* we cannot process subdirs! */ - - DBGPRINTF("fen_DirSearchFiles found matching file '%s' \n", file); - - /* Get correct dirindex from basedir! */ - dirIdx = dirsFindDir(basedir); - if (dirIdx == -1) { - /* Add dir to table and create watch */ - CHKiRet(dirsAdd(basedir, &dirIdx)); - DBGPRINTF("fen_DirSearchFiles adding new dir '%s' to dirs table idx %d\n", - basedir, dirIdx); -// fen_processEventDir(NULL, dirIdx, 0); /* Monitor child directory as well */ - } - - /* Search for file index here */ - ftIdx = fileTableSearchNoWildcard(&dirs[dirIdx].active, (uchar*)basefilename); - if(ftIdx >= 0) { - DBGPRINTF("fen_DirSearchFiles file '%s' idx %d already being monitored ... \n", - file, ftIdx); - } else { - DBGPRINTF("fen_DirSearchFiles setup new monitor for dynamic file '%s' \n", file); - - /* duplicate listener firs */ - pLstnNew = pLstn; - localRet = lstnDup(&pLstnNew, basefilename, dirs[dirIdx].dirName); - if(localRet != RS_RET_OK) { - DBGPRINTF("fen_DirSearchFiles failed to duplicate listener for '%s' " - "with iRet %d\n", localRet, pLstn->pszFileName); - ABORT_FINALIZE(localRet); - } - - // Create FileInfo struct - pLstnNew->pfinf = malloc(sizeof(struct fileinfo)); - if (pLstnNew->pfinf == NULL) { - LogError(errno, RS_RET_IO_ERROR, "fen_DirSearchFiles alloc memory " - "for fileinfo failed "); - ABORT_FINALIZE(RS_RET_IO_ERROR); - } - if ((pLstnNew->pfinf->fobj.fo_name = strdup((char*)pLstnNew->pszFileName)) == NULL) { - LogError(errno, RS_RET_IO_ERROR, "fen_DirSearchFiles alloc memory " - "for strdup failed "); - free(pLstnNew->pfinf); - pLstnNew->pfinf = NULL; - ABORT_FINALIZE(RS_RET_IO_ERROR); - } - /* Event types to watch. */ - pLstnNew->pfinf->events = FILE_MODIFIED; - pLstnNew->pfinf->port = glport; - - /* Add Listener to configured dirs tab */ - dirsAddFile(pLstnNew, ACTIVE_FILE); - - DBGPRINTF("fen_DirSearchFiles duplicated listener for '%s/%s' \n", - pLstnNew->pszDirName, pLstnNew->pszBaseName); - - /* Trigger file processing */ - fen_processEventFile(NULL, pLstnNew, FILE_MODIFIED, dirIdx); - } - - } - globfree(&files); - } else { - DBGPRINTF("fen_DirSearchFiles found ZERO matches with pattern '%s' \n", pLstn->pszFileName); - } - -finalize_it: - RETiRet; -} - -/* function not used yet, will be needed for wildcards later */ -rsRetVal -fen_processEventDir(struct file_obj* fobjp, int dirIdx, int revents) -{ - int iListIdx; - struct stat statFile; - sbool hasWildcard; - DEFiRet; - - DBGPRINTF("fen_processEventDir '%s' (Configured %d)", dirs[dirIdx].dirName, - dirs[dirIdx].configured.currMax); - - // Use FileObj from dirinfo if NULL - if (fobjp== NULL) - fobjp = &dirs[dirIdx].pfinf->fobj; - - /* Port needs to be reassociated */ - dirs[dirIdx].bPortAssociated = 0; - - if (revents) { - fen_printevent(revents); - DBGPRINTF("\n"); - DBGPRINTF("fen_processEventDir DIR EVENTS needs to be processed for '%s'('%s')\n", - fobjp->fo_name, dirs[dirIdx].dirName); - - /* a file was modified */ - if (revents & FILE_MODIFIED) { - /* LOOP through configured Listeners */ - for(iListIdx = 0; iListIdx < dirs[dirIdx].configured.currMax; iListIdx++) { - hasWildcard = ( dirs[dirIdx].hasWildcard || - dirs[dirIdx].configured.listeners[iListIdx].pLstn->hasWildcard - ? TRUE : FALSE); - if (hasWildcard == 1){ - /* Handle Wildcard files */ - fen_DirSearchFiles( dirs[dirIdx].configured.listeners[iListIdx].pLstn, - dirIdx); - } else { - /* Handle fixed configured files */ - if (dirs[dirIdx].configured.listeners[iListIdx].pLstn->bPortAssociated == 0) { - DBGPRINTF("fen_processEventDir Listener for %s needs to be checked\n", - dirs[dirIdx].configured.listeners[iListIdx].pLstn->pszFileName); - /* Need to check if listener file was created! */ - fen_processEventFile(NULL, - dirs[dirIdx].configured.listeners[iListIdx].pLstn, - FILE_MODIFIED - /*dirs[dirIdx].configured.listeners[iListIdx].pLstn->pfinf->events*/, - dirIdx); - } else { - DBGPRINTF("fen_processEventDir Listener for %s already associated\n", - dirs[dirIdx].configured.listeners[iListIdx].pLstn->pszFileName); - } - } - } - } - } else { - DBGPRINTF("\n"); - } - - /* Check if dir exists */ - if (stat(fobjp->fo_name, &statFile) == 0 && S_ISDIR(statFile.st_mode)) { - DBGPRINTF("fen_processEventDir '%s'('%s') is a valid directory, associate port\n" - , fobjp->fo_name, dirs[dirIdx].dirName, errno); - } else { - DBGPRINTF("fen_processEventDir Failed to stat directory: '%s'('%s') - errno %d, removing\n" - , fobjp->fo_name, dirs[dirIdx].dirName, errno); - fen_removeDir(dirIdx); /* Remove dir */ - ABORT_FINALIZE(RS_RET_FILE_NO_STAT); - } - - /* Register file event */ - fobjp->fo_atime = statFile.st_atim; - fobjp->fo_mtime = statFile.st_mtim; - fobjp->fo_ctime = statFile.st_ctim; - if (port_associate(glport, PORT_SOURCE_FILE, (uintptr_t)fobjp, dirs[dirIdx].pfinf->events, - (void *)dirIdx) == -1) { - /* Add error processing as required, file may have been deleted/moved. */ - LogError(errno, RS_RET_SYS_ERR, "fen_processEventDir: Failed to associated port " - "for directory: '%s'('%s') - errno %d\n", fobjp->fo_name, dirs[dirIdx].dirName, errno); - ABORT_FINALIZE(RS_RET_SYS_ERR); - } else { - /* Port successfull listening now*/ - DBGPRINTF("fen_processEventDir associated port for dir '%s'('%s')\n" - , fobjp->fo_name, dirs[dirIdx].dirName); - dirs[dirIdx].bPortAssociated = 1; - } -finalize_it: - RETiRet; - -} - -/* Setup directory watches, based on user config */ -static void -fen_setupDirWatches(void) -{ - int i; - for(i = 0 ; i < currMaxDirs ; ++i) { - if ( dirs[i].bPortAssociated == 0 && - dirs[i].dirName != NULL /* Don't check deleted dirs */ ){ - fen_processEventDir(NULL, i, 0);/* no events on init*/ - } - } -} - -/* Setup static file watches, based on user config */ -static void -fen_setupFileWatches(void) -{ - /* Listener helper*/ - lstn_t *pLstn; - int dirIdx; - - /* Additional check for not associated files */ - for(pLstn = runModConf->pRootLstn ; pLstn != NULL ; pLstn = pLstn->next) { - /* Search for dirIdx in case directory has wildcard */ - dirIdx = dirsFindDir(pLstn->pszDirName); - - if ( pLstn->bPortAssociated == 0 && - pLstn->hasWildcard == 0 /* WildCard File Watches are Setup in EventDir */ && - (dirIdx >= 0 && dirs[dirIdx].hasWildcard == FALSE) - ){ - /* Check if file exists now */ - fen_processEventFile(NULL, pLstn, pLstn->pfinf->events, dirIdx); - } - } -} +/* https://docs.oracle.com/cd/E19253-01/816-5168/port-get-3c/index.html */ static rsRetVal do_fen(void) { - int bHadFileData; /* were there at least one file with data during this run? */ port_event_t portEvent; struct timespec timeout; - lstn_t *pLstn; /* Listener helper*/ - struct file_obj *fobjp = NULL; /* Helper object */ - struct stat statFile; DEFiRet; - rsRetVal iRetTmp = RS_RET_OK; + //rsRetVal iRetTmp = RS_RET_OK; - /* Set port timeout to 1 second. We need to checkfor unmonitored files during meantime */ - timeout.tv_sec = 1; + /* Set port timeout to 1 second. We need to check for unmonitored files during meantime */ + // TODO: do we need this timeout at all for equality to old code? + // TODO: do we need it to support the timeout feature! + timeout.tv_sec = 300; timeout.tv_nsec = 0; /* create port instance */ - if ((glport = port_create()) == -1) { + if((glport = port_create()) == -1) { LogError(errno, RS_RET_FEN_INIT_FAILED, "do_fen INIT Port failed "); return RS_RET_FEN_INIT_FAILED; } - /* create port instance */ - CHKiRet(dirsInit()); - - /* Loop through all configured listeners */ - for(pLstn = runModConf->pRootLstn ; pLstn != NULL ; pLstn = pLstn->next) { - if(pLstn->masterLstn == NULL) { - DBGPRINTF("do_fen process '%s' in '%s'\n", pLstn->pszBaseName, pLstn->pszDirName); - // Create FileInfo struct - pLstn->pfinf = malloc(sizeof(struct fileinfo)); - if (pLstn->pfinf == NULL) { - LogError(errno, RS_RET_FEN_INIT_FAILED, "do_fen: alloc memory " - "for fileinfo failed "); - ABORT_FINALIZE(RS_RET_FEN_INIT_FAILED); - } - if ((pLstn->pfinf->fobj.fo_name = strdup((char*)pLstn->pszFileName)) == NULL) { - LogError(errno, RS_RET_FEN_INIT_FAILED, "do_fen: alloc memory " - "for strdup failed "); - free(pLstn->pfinf); - pLstn->pfinf = NULL; - ABORT_FINALIZE(RS_RET_FEN_INIT_FAILED); - } - - /* Event types to watch. */ - pLstn->pfinf->events = FILE_MODIFIED; - /* Not needed/working |FILE_DELETE|FILE_RENAME_TO|FILE_RENAME_FROM;*/ - pLstn->pfinf->port = glport; - } - - /* Add Listener to configured dirs tab */ - dirsAddFile(pLstn, CONFIGURED_FILE); - } - - /* Init File watches ONCE */ - fen_setupFileWatches(); + /* do watch initialization */ + fs_node_walk(runModConf->conf_tree, poll_tree); DBGPRINTF("do_fen ENTER monitoring loop \n"); while(glbl.GetGlobalInputTermState() == 0) { DBGPRINTF("do_fen loop begin... \n"); + /* Loop through events, if there are any */ + while (!port_get(glport, &portEvent, &timeout)) { // wie inotify-wait + DBGPRINTF("do_fen: received port event with "); + fen_printevent((int) portEvent.portev_events); + DBGPRINTF("\n"); + if(portEvent.portev_source != PORT_SOURCE_FILE) { + LogError(errno, RS_RET_SYS_ERR, "do_fen: Event from unexpected source " + ": %d\n", portEvent.portev_source); + continue; + } + act_obj_t *const act = (act_obj_t*) portEvent.portev_user; + DBGPRINTF("do_fen event received: deleted %d, is_file %d, name '%s' foname '%s'\n", + act->is_deleted, act->edge->is_file, act->name, + ((struct file_obj*)portEvent.portev_object)->fo_name); + if(act->is_deleted) { + free(act->name); + free(act); + continue; + } - /* Check for not associated directories and add dir watches */ - fen_setupDirWatches(); + /* we need to re-associate the object */ + act->bPortAssociated = 0; + fen_setupWatch(act); - /* Loop through events, if there are any */ - while (!port_get(glport, &portEvent, &timeout)) { - switch (portEvent.portev_source) { - case PORT_SOURCE_FILE: - /* check if file obj is DIR or FILE */ - fobjp = (struct file_obj*) portEvent.portev_object; - DBGPRINTF("do_fen event received for '%s', processing ... \n", - fobjp->fo_name); - - /* Check if we habe a DIR or FILE */ - if (stat(fobjp->fo_name, &statFile) == 0 && S_ISDIR(statFile.st_mode)) { - fen_processEventDir(fobjp, (int)portEvent.portev_user, - portEvent.portev_events); - } else { - /* Call file events event handler */ - fen_processEventFile(fobjp, (lstn_t*)portEvent.portev_user, - portEvent.portev_events, -1 /* Unknown diridx */); - } - break; - default: - LogError(errno, RS_RET_SYS_ERR, "do_fen: Event from unexpected source " - ": %d\n", portEvent.portev_source); + if(act->edge->is_file) { + pollFile(act); + } else { + // curr: fs_node_walk(act->edge->parent, poll_tree); + fs_node_walk(act->edge->node, poll_tree); } } - - DBGPRINTF("do_fen loop end... \n"); } - DBGPRINTF("do_fen EXIT monitoring loop \n"); -finalize_it: - /* - * close port, will de-activate all file events watches associated - * with the port. - */ + /* close port, will de-activate all file events watches associated + * with the port. + */ close(glport); - - /* Free memory now */ - for(pLstn = runModConf->pRootLstn ; pLstn != NULL ; pLstn = pLstn->next) { - free(pLstn->pfinf->fobj.fo_name); - free(pLstn->pfinf); - pLstn->pfinf = NULL; - } - RETiRet; } #else /* #if OS_SOLARIS */ @@ -3212,7 +2154,6 @@ do_fen(void) "platform does not support fen"); return RS_RET_NOT_IMPLEMENTED; } - #endif /* #if OS_SOLARIS */ @@ -3263,17 +2204,18 @@ ENDwillRun * iRet as that makes matters worse (at least we can try persisting the others...). * rgerhards, 2008-02-13 */ -static rsRetVal -persistStrmState(lstn_t *pLstn) +static rsRetVal ATTR_NONNULL() +persistStrmState(act_obj_t *const act) { DEFiRet; strm_t *psSF = NULL; /* state file (stream) */ size_t lenDir; uchar statefile[MAXFNAME]; - uchar *const statefn = getStateFileName(pLstn, statefile, sizeof(statefile), NULL); - DBGPRINTF("persisting state for '%s' to file '%s'\n", - pLstn->pszFileName, statefn); + const instanceConf_t *const inst = act->edge->instarr[0];// TODO: same file, multiple instances? + + uchar *const statefn = getStateFileName(act, statefile, sizeof(statefile)); + DBGPRINTF("persisting state for '%s' to file '%s'\n", act->name, statefn); CHKiRet(strm.Construct(&psSF)); lenDir = ustrlen(glbl.GetWorkDir()); if(lenDir > 0) @@ -3281,10 +2223,10 @@ persistStrmState(lstn_t *pLstn) CHKiRet(strm.SettOperationsMode(psSF, STREAMMODE_WRITE_TRUNC)); CHKiRet(strm.SetsType(psSF, STREAMTYPE_FILE_SINGLE)); CHKiRet(strm.SetFName(psSF, statefn, strlen((char*) statefn))); - CHKiRet(strm.SetFileNotFoundError(psSF, pLstn->fileNotFoundError)); + CHKiRet(strm.SetFileNotFoundError(psSF, inst->fileNotFoundError)); CHKiRet(strm.ConstructFinalize(psSF)); - CHKiRet(strm.Serialize(pLstn->pStrm, psSF)); + CHKiRet(strm.Serialize(act->pStrm, psSF)); CHKiRet(strm.Flush(psSF)); CHKiRet(strm.Destruct(&psSF)); @@ -3308,11 +2250,6 @@ finalize_it: */ BEGINafterRun CODESTARTafterRun - while(runModConf->pRootLstn != NULL) { - /* Note: lstnDel() reasociates root! */ - lstnDel(runModConf->pRootLstn); - } - if(pInputName != NULL) prop.Destruct(&pInputName); ENDafterRun @@ -3337,28 +2274,9 @@ CODESTARTmodExit objRelease(prop, CORE_COMPONENT); objRelease(ruleset, CORE_COMPONENT); -# if defined(HAVE_INOTIFY_INIT) || (defined(OS_SOLARIS) && defined (HAVE_PORT_SOURCE_FILE)) - int i; - /* we use these vars only in inotify mode */ - if(dirs != NULL) { - /* Free dirNames */ - for(i = 0 ; i < currMaxDirs ; ++i) { - free(dirs[i].dirName); - free(dirs[i].dirNameBfWildCard); -# if defined(OS_SOLARIS) && defined (HAVE_PORT_SOURCE_FILE) - free(dirs[i].pfinf->fobj.fo_name); - free(dirs[i].pfinf); -# endif - } - free(dirs->active.listeners); - free(dirs->configured.listeners); - free(dirs); - } -# endif /* #if defined(HAVE_INOTIFY_INIT) || (defined(OS_SOLARIS) && defined (HAVE_PORT_SOURCE_FILE)) --- */ - -#ifdef HAVE_INOTIFY_INIT + #ifdef HAVE_INOTIFY_INIT free(wdmap); -#endif + #endif ENDmodExit diff --git a/tests/Makefile.am b/tests/Makefile.am index 75ed68c1e..2283066ac 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -761,6 +761,7 @@ TESTS += \ imfile-error-not-repeated.sh \ imfile-truncate.sh \ imfile-readmode2.sh \ + imfile-readmode2-polling.sh \ imfile-readmode2-with-persists-data-during-stop.sh \ imfile-readmode2-with-persists.sh \ imfile-endregex.sh \ @@ -776,17 +777,15 @@ TESTS += \ imfile-wildcards.sh \ imfile-wildcards-dirs.sh \ imfile-wildcards-dirs2.sh \ + imfile-wildcards-dirs-multi.sh \ + imfile-wildcards-dirs-multi2.sh \ + imfile-wildcards-dirs-multi3.sh \ + imfile-wildcards-dirs-multi4.sh \ + imfile-wildcards-dirs-multi5.sh \ + imfile-wildcards-dirs-multi5-polling.sh \ + imfile-rename-while-stopped.sh \ imfile-rename.sh -# TEMPORARILY disable some imfile tests because refactoring -# is needed to get them to work 100% all the time. -# alorbach, 2018-01-05 -# imfile-wildcards-dirs-multi.sh \ -# imfile-wildcards-dirs-multi2.sh \ -# imfile-wildcards-dirs-multi3.sh \ -# imfile-wildcards-dirs-multi4.sh \ -# - if HAVE_VALGRIND TESTS += \ imfile-basic-vg.sh \ @@ -1440,6 +1439,7 @@ EXTRA_DIST= \ random.sh \ testsuites/random.conf \ imfile-readmode2.sh \ + imfile-readmode2-polling.sh \ imfile-readmode2-vg.sh \ imfile-readmode2-with-persists-data-during-stop.sh \ imfile-readmode2-with-persists.sh \ @@ -1475,8 +1475,10 @@ EXTRA_DIST= \ imfile-wildcards-dirs-multi2.sh \ imfile-wildcards-dirs-multi3.sh \ imfile-wildcards-dirs-multi4.sh \ + imfile-wildcards-dirs-multi5.sh \ + imfile-wildcards-dirs-multi5-polling.sh \ + imfile-rename-while-stopped.sh \ imfile-rename.sh \ - testsuites/imfile-wildcards.conf \ testsuites/imfile-wildcards-simple.conf \ testsuites/imfile-wildcards-dirs.conf \ testsuites/imfile-wildcards-dirs-multi.conf \ diff --git a/tests/diag.sh b/tests/diag.sh index 616acdafe..1e91a4ee6 100755 --- a/tests/diag.sh +++ b/tests/diag.sh @@ -578,6 +578,36 @@ case $1 in . $srcdir/diag.sh error-exit 1 fi ;; + 'wait-file-lines') + # $2 filename, $3 expected nbr of lines, $4 nbr of tries + if [ "$4" == "" ]; then + timeoutend=1 + else + timeoutend=$4 + fi + timecounter=0 + + while [ $timecounter -lt $timeoutend ]; do + let timecounter=timecounter+1 + + count=$(wc -l < rsyslog.out.log) + if [ $count -eq $3 ]; then + echo wait-file-lines success, have $3 lines + break + else + if [ "x$timecounter" == "x$timeoutend" ]; then + echo wait-file-lines failed, expected $3 got $count + . $srcdir/diag.sh shutdown-when-empty + . $srcdir/diag.sh wait-shutdown + . $srcdir/diag.sh error-exit 1 + else + echo wait-file-lines not yet there, currently $count lines + ./msleep 200 + fi + fi + done + unset count + ;; 'content-check-with-count') # content check variables for Timeout if [ "x$4" == "x" ]; then @@ -588,12 +618,11 @@ case $1 in timecounter=0 while [ $timecounter -lt $timeoutend ]; do -# echo content-check-with-count loop $timecounter let timecounter=timecounter+1 count=$(cat rsyslog.out.log | grep -F "$2" | wc -l) - if [ "x$count" == "x$3" ]; then + if [ $count -eq $3 ]; then echo content-check-with-count success, \"$2\" occured $3 times break else @@ -601,12 +630,12 @@ case $1 in . $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd . $srcdir/diag.sh wait-shutdown # Shutdown rsyslog instance on error - echo content-check-with-count failed, expected \"$2\" to occure $3 times, but found it $count times + echo content-check-with-count failed, expected \"$2\" to occur $3 times, but found it $count times echo file rsyslog.out.log content is: - cat rsyslog.out.log + sort < rsyslog.out.log . $srcdir/diag.sh error-exit 1 else - echo content-check-with-count failed, trying again ... + echo content-check-with-count have $count, wait for $3 times $2... ./msleep 1000 fi fi diff --git a/tests/imfile-discard-truncated-line.sh b/tests/imfile-discard-truncated-line.sh index 31ba1a29e..cf2cf3abe 100755 --- a/tests/imfile-discard-truncated-line.sh +++ b/tests/imfile-discard-truncated-line.sh @@ -54,7 +54,7 @@ printf 'HEADER msgnum:0 HEADER msgnum:1 HEADER msgnum:2 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\\\\n msgnum:3 bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\\\\n msgnum:4 ccccccc HEADER msgnum:6 eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee\\\\n msgnum:7 ffffffffffffffffffffffffffffffffffffffffffff\\\\n msgnum:8 ggggggg -HEADER msgnum:9\n' | cmp -b rsyslog.out.log +HEADER msgnum:9\n' | cmp - rsyslog.out.log if [ ! $? -eq 0 ]; then echo "invalid multiline message generated, rsyslog.out.log is:" cat rsyslog.out.log diff --git a/tests/imfile-endregex-save-lf-persist.sh b/tests/imfile-endregex-save-lf-persist.sh index 808e98d8e..260aeb5cd 100755 --- a/tests/imfile-endregex-save-lf-persist.sh +++ b/tests/imfile-endregex-save-lf-persist.sh @@ -47,7 +47,7 @@ echo 'END OF TEST' >> rsyslog.input printf 'HEADER msgnum:0\\\\n msgnum:1\\\\n msgnum:2 HEADER msgnum:3 -HEADER msgnum:4\\\\n msgnum:5\n' | cmp -b rsyslog.out.log +HEADER msgnum:4\\\\n msgnum:5\n' | cmp - rsyslog.out.log if [ ! $? -eq 0 ]; then echo "invalid multiline message generated, rsyslog.out.log is:" cat rsyslog.out.log diff --git a/tests/imfile-endregex-save-lf.sh b/tests/imfile-endregex-save-lf.sh index 67a51c66f..5ba03250a 100755 --- a/tests/imfile-endregex-save-lf.sh +++ b/tests/imfile-endregex-save-lf.sh @@ -41,7 +41,7 @@ echo 'END OF TEST' >> rsyslog.input . $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages . $srcdir/diag.sh wait-shutdown # we need to wait until rsyslogd is finished! -printf 'HEADER msgnum:0\\\\n msgnum:1\\\\n msgnum:2\n' | cmp -b rsyslog.out.log +printf 'HEADER msgnum:0\\\\n msgnum:1\\\\n msgnum:2\n' | cmp - rsyslog.out.log if [ ! $? -eq 0 ]; then echo "invalid multiline message generated, rsyslog.out.log is:" cat rsyslog.out.log diff --git a/tests/imfile-endregex-timeout-none.sh b/tests/imfile-endregex-timeout-none.sh index 6d5b2683a..a639c9a59 100755 --- a/tests/imfile-endregex-timeout-none.sh +++ b/tests/imfile-endregex-timeout-none.sh @@ -2,7 +2,7 @@ # This is part of the rsyslog testbench, licensed under ASL 2.0 echo ====================================================================== echo [imfile-endregex-timeout-none.sh] -. $srcdir/diag.sh check-inotify-only +. $srcdir/diag.sh check-inotify . $srcdir/diag.sh init . $srcdir/diag.sh generate-conf . $srcdir/diag.sh add-conf ' @@ -42,7 +42,7 @@ echo 'END OF TEST' >> rsyslog.input . $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages . $srcdir/diag.sh wait-shutdown # we need to wait until rsyslogd is finished! -printf 'HEADER msgnum:0\\\\n msgnum:1\\\\n msgnum:2\\\\n msgnum:3\n' | cmp -b rsyslog.out.log +printf 'HEADER msgnum:0\\\\n msgnum:1\\\\n msgnum:2\\\\n msgnum:3\n' | cmp - rsyslog.out.log if [ ! $? -eq 0 ]; then echo "invalid multiline message generated, rsyslog.out.log is:" cat rsyslog.out.log diff --git a/tests/imfile-endregex-timeout-with-shutdown.sh b/tests/imfile-endregex-timeout-with-shutdown.sh index 5644b6bf7..bd9221c69 100755 --- a/tests/imfile-endregex-timeout-with-shutdown.sh +++ b/tests/imfile-endregex-timeout-with-shutdown.sh @@ -61,7 +61,7 @@ echo ' msgnum:5 printf 'HEADER msgnum:0\\\\n msgnum:1 HEADER msgnum:2\\\\n msgnum:3\\\\n msgnum:4 -HEADER msgnum:5\\\\n msgnum:6\n' | cmp -b rsyslog.out.log +HEADER msgnum:5\\\\n msgnum:6\n' | cmp - rsyslog.out.log if [ ! $? -eq 0 ]; then echo "invalid multiline message generated, rsyslog.out.log is:" cat rsyslog.out.log diff --git a/tests/imfile-endregex-timeout.sh b/tests/imfile-endregex-timeout.sh index 13786bed7..249895692 100755 --- a/tests/imfile-endregex-timeout.sh +++ b/tests/imfile-endregex-timeout.sh @@ -46,7 +46,7 @@ echo 'END OF TEST' >> rsyslog.input . $srcdir/diag.sh wait-shutdown # we need to wait until rsyslogd is finished! printf 'HEADER msgnum:0\\\\n msgnum:1 -HEADER msgnum:2\\\\n msgnum:3\n' | cmp -b rsyslog.out.log +HEADER msgnum:2\\\\n msgnum:3\n' | cmp - rsyslog.out.log if [ ! $? -eq 0 ]; then echo "invalid multiline message generated, rsyslog.out.log is:" cat rsyslog.out.log diff --git a/tests/imfile-file-not-found-error.sh b/tests/imfile-file-not-found-error.sh index eab0e7cea..268e34fa4 100755 --- a/tests/imfile-file-not-found-error.sh +++ b/tests/imfile-file-not-found-error.sh @@ -1,7 +1,7 @@ #!/bin/bash # add 2017-04-28 by Pascal Withopf, released under ASL 2.0 echo [imfile-file-not-found-error.sh] -. $srcdir/diag.sh check-inotify-only +. $srcdir/diag.sh check-inotify . $srcdir/diag.sh init . $srcdir/diag.sh generate-conf . $srcdir/diag.sh add-conf ' @@ -35,7 +35,7 @@ fi printf 'testmessage1 testmessage2 -testmessage3\n' | cmp -b rsyslog.out.log +testmessage3\n' | cmp - rsyslog.out.log if [ ! $? -eq 0 ]; then echo "invalid response generated, rsyslog.out.log is:" cat rsyslog.out.log diff --git a/tests/imfile-readmode2-with-persists.sh b/tests/imfile-readmode2-with-persists.sh index 9db15b559..c8083201c 100755 --- a/tests/imfile-readmode2-with-persists.sh +++ b/tests/imfile-readmode2-with-persists.sh @@ -20,6 +20,10 @@ sleep 1 echo stopping rsyslog . $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages . $srcdir/diag.sh wait-shutdown # we need to wait until rsyslogd is finished! + +echo spool: +ls -l test-spool + echo restarting rsyslog . $srcdir/diag.sh startup imfile-readmode2-with-persists.conf echo restarted rsyslog, continuing with test diff --git a/tests/imfile-readmode2.sh b/tests/imfile-readmode2.sh index ff860ecbe..c0e074a9c 100755 --- a/tests/imfile-readmode2.sh +++ b/tests/imfile-readmode2.sh @@ -1,10 +1,30 @@ #!/bin/bash # This is part of the rsyslog testbench, licensed under ASL 2.0 -echo ====================================================================== -echo [imfile-readmode2.sh] -. $srcdir/diag.sh check-inotify +#export RSYSLOG_DEBUG="debug nologfuncflow noprintmutexaction stdout" +#export RSYSLOG_DEBUGLOG="log" + . $srcdir/diag.sh init -. $srcdir/diag.sh startup imfile-readmode2.conf +. $srcdir/diag.sh check-inotify +. $srcdir/diag.sh generate-conf +. $srcdir/diag.sh add-conf ' +global( debug.whitelist="on" + debug.files=["imfile.c"]) + +module(load="../plugins/imfile/.libs/imfile") + +input(type="imfile" File="./rsyslog.input" Tag="file:" ReadMode="2") +#input(type="imfile" File="./rsyslog.input.*" Tag="file:" ReadMode="2") + +template(name="outfmt" type="list") { + constant(value="HEADER ") + property(name="msg" format="json") + constant(value="\n") +} + +if $msg contains "msgnum:" then + action(type="omfile" file="rsyslog.out.log" template="outfmt" + ) ' +. $srcdir/diag.sh startup # write the beginning of the file echo 'msgnum:0 @@ -20,22 +40,23 @@ echo 'msgnum:3 echo 'msgnum:5' >> rsyslog.input # this one shouldn't be written to the output file because of ReadMode 2 # give it time to finish -sleep 1 +#echo you are in bash! ctl-d to exit +#bash . $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages . $srcdir/diag.sh wait-shutdown # we need to wait until rsyslogd is finished! # give it time to write the output file + sleep 1 +#cat log ## check if we have the correct number of messages NUMLINES=$(grep -c HEADER ./rsyslog.out.log 2>/dev/null) if [ -z $NUMLINES ]; then - echo "ERROR: expecting at least a match for HEADER, maybe rsyslog.out.log wasn't even written?" - cat ./rsyslog.out.log - exit 1 + echo "ERROR: expecting at least a match for HEADER, maybe rsyslog.out.log wasn't even written?" cat ./rsyslog.out.log exit 1 else if [ ! $NUMLINES -eq 3 ]; then echo "ERROR: expecting 3 headers, got $NUMLINES" diff --git a/tests/imfile-rename-while-stopped.sh b/tests/imfile-rename-while-stopped.sh new file mode 100755 index 000000000..79b1b455e --- /dev/null +++ b/tests/imfile-rename-while-stopped.sh @@ -0,0 +1,33 @@ +#!/bin/bash +# This is part of the rsyslog testbench, licensed under GPLv3 +export TESTMESSAGES=10000 +export RETRIES=10 +export TESTMESSAGESFULL=19999 +echo [imfile-rename.sh] +. $srcdir/diag.sh check-inotify-only +. $srcdir/diag.sh init + +# generate input file first. +./inputfilegen -m $TESTMESSAGES > rsyslog.input.1.log +ls -li rsyslog.input* + +. $srcdir/diag.sh startup imfile-wildcards-simple.conf +. $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +. $srcdir/diag.sh wait-shutdown # we need to wait until rsyslogd is finished! + +# Move to another filename +mv rsyslog.input.1.log rsyslog.input.2.log + +# generate some more input into moved file +./inputfilegen -m $TESTMESSAGES -i $TESTMESSAGES >> rsyslog.input.2.log +ls -li rsyslog.input* +echo ls test-spool: +ls -l test-spool + +. $srcdir/diag.sh startup imfile-wildcards-simple.conf +. $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +. $srcdir/diag.sh wait-shutdown # we need to wait until rsyslogd is finished! + +. $srcdir/diag.sh seq-check 0 $TESTMESSAGESFULL +wc rsyslog.out.log +. $srcdir/diag.sh exit diff --git a/tests/imfile-rename.sh b/tests/imfile-rename.sh index 8646f5580..7a720169a 100755 --- a/tests/imfile-rename.sh +++ b/tests/imfile-rename.sh @@ -1,6 +1,7 @@ #!/bin/bash # This is part of the rsyslog testbench, licensed under GPLv3 export TESTMESSAGES=10000 +export RETRIES=10 export TESTMESSAGESFULL=19999 echo [imfile-rename.sh] . $srcdir/diag.sh check-inotify-only @@ -13,17 +14,22 @@ ls -l rsyslog.input* . $srcdir/diag.sh startup imfile-wildcards-simple.conf # sleep a little to give rsyslog a chance to begin processing -sleep 5 + +. $srcdir/diag.sh wait-file-lines rsyslog.out.log $TESTMESSAGES $RETRIES # Move to another filename mv rsyslog.input.1.log rsyslog.input.2.log +./msleep 500 # generate some more input into moved file ./inputfilegen -m $TESTMESSAGES -i $TESTMESSAGES >> rsyslog.input.2.log ls -l rsyslog.input* +echo ls test-spool: +ls -l test-spool +./msleep 500 -# sleep a little to give rsyslog a chance to begin processing -sleep 5 +let msgcount="2* $TESTMESSAGES" +. $srcdir/diag.sh wait-file-lines rsyslog.out.log $msgcount $RETRIES . $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages . $srcdir/diag.sh wait-shutdown # we need to wait until rsyslogd is finished! diff --git a/tests/imfile-truncate-line.sh b/tests/imfile-truncate-line.sh index 5c17f3226..72587c659 100755 --- a/tests/imfile-truncate-line.sh +++ b/tests/imfile-truncate-line.sh @@ -56,7 +56,7 @@ HEADER msgnum:2 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\\\\n msgnum:3 bbbbb HEADER ccccccccccccccccccccccccccccccccccccc\\\\n msgnum:5 dddddddddddddddddddddddddddddddddddddddddddd HEADER msgnum:6 eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee\\\\n msgnum:7 ffffffffffffffffffffffffffffffffffffffffffff\\\\n msgnum:8 ggggggg HEADER ggggggggggggggggggggggggggggggggggggg -HEADER msgnum:9\n' | cmp -b - rsyslog.out.log +HEADER msgnum:9\n' | cmp - rsyslog.out.log if [ ! $? -eq 0 ]; then echo "invalid multiline message generated, rsyslog.out.log is:" cat rsyslog.out.log diff --git a/tests/imfile-wildcards-dirs-multi.sh b/tests/imfile-wildcards-dirs-multi.sh index f5295beee..ea10ba780 100755 --- a/tests/imfile-wildcards-dirs-multi.sh +++ b/tests/imfile-wildcards-dirs-multi.sh @@ -3,7 +3,7 @@ export IMFILEINPUTFILES="10" export IMFILEINPUTFILESSTEPS="5" #export IMFILEINPUTFILESALL=$(($IMFILEINPUTFILES * $IMFILEINPUTFILESSTEPS)) -export IMFILECHECKTIMEOUT="5" +export IMFILECHECKTIMEOUT="20" . $srcdir/diag.sh init . $srcdir/diag.sh check-inotify-only # generate input files first. Note that rsyslog processes it as diff --git a/tests/imfile-wildcards-dirs-multi2.sh b/tests/imfile-wildcards-dirs-multi2.sh index 59d844097..01413e559 100755 --- a/tests/imfile-wildcards-dirs-multi2.sh +++ b/tests/imfile-wildcards-dirs-multi2.sh @@ -3,9 +3,8 @@ export IMFILEINPUTFILES="1" export IMFILEINPUTFILESSTEPS="5" #export IMFILEINPUTFILESALL=$(($IMFILEINPUTFILES * $IMFILEINPUTFILESSTEPS)) -export IMFILECHECKTIMEOUT="5" +export IMFILECHECKTIMEOUT="20" . $srcdir/diag.sh init -. $srcdir/diag.sh check-inotify-only # generate input files first. Note that rsyslog processes it as # soon as it start up (so the file should exist at that point). diff --git a/tests/imfile-wildcards-dirs-multi3.sh b/tests/imfile-wildcards-dirs-multi3.sh index 23c2139bd..0fecffa81 100755 --- a/tests/imfile-wildcards-dirs-multi3.sh +++ b/tests/imfile-wildcards-dirs-multi3.sh @@ -3,9 +3,8 @@ export IMFILEINPUTFILES="1" export IMFILEINPUTFILESSTEPS="5" #export IMFILEINPUTFILESALL=$(($IMFILEINPUTFILES * $IMFILEINPUTFILESSTEPS)) -export IMFILECHECKTIMEOUT="5" +export IMFILECHECKTIMEOUT="10" . $srcdir/diag.sh init -. $srcdir/diag.sh check-inotify-only # generate input files first. Note that rsyslog processes it as # soon as it start up (so the file should exist at that point). @@ -13,6 +12,7 @@ for i in `seq 1 $IMFILEINPUTFILES`; do echo "Make rsyslog.input.dir$i" mkdir rsyslog.input.dir$i + echo created! done # Start rsyslog now before adding more files @@ -30,6 +30,7 @@ do mkdir rsyslog.input.dir$i/dir$j/testdir mkdir rsyslog.input.dir$i/dir$j/testdir/subdir$j ./inputfilegen -m 1 > rsyslog.input.dir$i/dir$j/testdir/subdir$j/file.logfile + ls -l rsyslog.input.dir$i/dir$j/testdir/subdir$j/file.logfile done ls -d rsyslog.input.* diff --git a/tests/imfile-wildcards-dirs-multi4.sh b/tests/imfile-wildcards-dirs-multi4.sh index a8121a71b..454ae7c4d 100755 --- a/tests/imfile-wildcards-dirs-multi4.sh +++ b/tests/imfile-wildcards-dirs-multi4.sh @@ -3,9 +3,9 @@ export IMFILEINPUTFILES="1" export IMFILEINPUTFILESSTEPS="5" #export IMFILEINPUTFILESALL=$(($IMFILEINPUTFILES * $IMFILEINPUTFILESSTEPS)) -export IMFILECHECKTIMEOUT="5" +export IMFILECHECKTIMEOUT="20" . $srcdir/diag.sh init -. $srcdir/diag.sh check-inotify-only +. $srcdir/diag.sh check-inotify # generate input files first. Note that rsyslog processes it as # soon as it start up (so the file should exist at that point). diff --git a/tests/imfile-wildcards-dirs-multi5-polling.sh b/tests/imfile-wildcards-dirs-multi5-polling.sh new file mode 100755 index 000000000..0689bd087 --- /dev/null +++ b/tests/imfile-wildcards-dirs-multi5-polling.sh @@ -0,0 +1,77 @@ +#!/bin/bash +# This is part of the rsyslog testbench, licensed under GPLv3 +export IMFILEINPUTFILES="8" #"8" +export IMFILEINPUTFILESSTEPS="5" #"5" +#export IMFILEINPUTFILESALL=$(($IMFILEINPUTFILES * $IMFILEINPUTFILESSTEPS)) +export IMFILECHECKTIMEOUT="5" +. $srcdir/diag.sh init +#. $srcdir/diag.sh check-inotify-only +# generate input files first. Note that rsyslog processes it as +# soon as it start up (so the file should exist at that point). + +# Start rsyslog now before adding more files +. $srcdir/diag.sh generate-conf +. $srcdir/diag.sh add-conf ' +global( debug.whitelist="on" + debug.files=["imfile.c"]) +# debug.files=["rainerscript.c", "ratelimit.c", "ruleset.c", "main Q", +# "msg.c", "../action.c", "imdiag.c"]) + +module(load="../plugins/imfile/.libs/imfile" mode="polling" pollingInterval="1") + +input(type="imfile" File="./rsyslog.input.dir?/*/*.logfile" + Tag="file:" Severity="error" Facility="local7" addMetadata="on") +input(type="imfile" File="./rsyslog.input.alt/alt*file" + Tag="file:" Severity="error" Facility="local7" addMetadata="on") + +template(name="outfmt" type="list") { + constant(value="HEADER ") + property(name="msg" format="json") + constant(value=", ") + property(name="$!metadata!filename") + constant(value="\n") +} + +if $msg contains "msgnum:" then + action( type="omfile" file="rsyslog.out.log" template="outfmt") +' + +# create first directory and file before startup, so ensure we will NOT +# get an initial inotify notify for it! +#mkdir rsyslog.input.alt +#./inputfilegen -m 1 > rsyslog.input.alt/altfile.logfile +#mkdir rsyslog.input.dir1 +# the following is INVALID, as this is a file, but must be a directory! +#./inputfilegen -m 1 > rsyslog.input.dir0 + +. $srcdir/diag.sh startup + +for j in `seq 1 $IMFILEINPUTFILESSTEPS`; +do + echo "Loop Num $j" + + for i in `seq 1 $IMFILEINPUTFILES`; + do + mkdir rsyslog.input.dir$i + mkdir rsyslog.input.dir$i/dir$i + ./inputfilegen -m 1 > rsyslog.input.dir$i/dir$i/file.logfile + done + + ls -d rsyslog.input.* + + # Check correct amount of input files each time + let IMFILEINPUTFILESALL=$(($IMFILEINPUTFILES * $j)) + . $srcdir/diag.sh content-check-with-count "HEADER msgnum:00000000:" $IMFILEINPUTFILESALL $IMFILECHECKTIMEOUT + + # Delete all but first! + for i in `seq 1 $IMFILEINPUTFILES`; + do + rm -rf rsyslog.input.dir$i/dir$i/file.logfile + rm -rf rsyslog.input.dir$i + done + +done + +. $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +. $srcdir/diag.sh wait-shutdown # we need to wait until rsyslogd is finished! +. $srcdir/diag.sh exit diff --git a/tests/imfile-wildcards-dirs-multi5.sh b/tests/imfile-wildcards-dirs-multi5.sh new file mode 100755 index 000000000..501db4338 --- /dev/null +++ b/tests/imfile-wildcards-dirs-multi5.sh @@ -0,0 +1,82 @@ +#!/bin/bash +# This is part of the rsyslog testbench, licensed under GPLv3 +export IMFILEINPUTFILES="8" #"8" +export IMFILEINPUTFILESSTEPS="5" #"5" +#export IMFILEINPUTFILESALL=$(($IMFILEINPUTFILES * $IMFILEINPUTFILESSTEPS)) +export IMFILECHECKTIMEOUT="20" +. $srcdir/diag.sh init +# generate input files first. Note that rsyslog processes it as +# soon as it start up (so the file should exist at that point). + +# Start rsyslog now before adding more files +. $srcdir/diag.sh generate-conf +. $srcdir/diag.sh add-conf ' +global( debug.whitelist="on" + debug.files=["imfile.c"]) + +module(load="../plugins/imfile/.libs/imfile" mode="inotify" pollingInterval="1") + +input(type="imfile" File="./rsyslog.input.dir?/*/*.logfile" + Tag="file:" Severity="error" Facility="local7" addMetadata="on") +input(type="imfile" File="./rsyslog.input.alt/alt*file" + Tag="file:" Severity="error" Facility="local7" addMetadata="on") + +template(name="outfmt" type="list") { + constant(value="HEADER ") + property(name="msg" format="json") + constant(value="'"'"', ") + property(name="$!metadata!filename") + constant(value="\n") +} + +if $msg contains "msgnum:" then + action( type="omfile" file="rsyslog.out.log" template="outfmt") + +#*.* action(type="omfile" file="rsyslog.debug") +' + +# create first directory and file before startup, so ensure we will NOT +# get an initial inotify notify for it! +mkdir rsyslog.input.alt +#./inputfilegen -m 1 > rsyslog.input.alt/altfile.logfile +mkdir rsyslog.input.dir1 +# the following is INVALID, as this is a file, but must be a directory! +./inputfilegen -m 1 > rsyslog.input.dir0 + +. $srcdir/diag.sh startup + +for j in `seq 1 $IMFILEINPUTFILESSTEPS`; +do + echo "Loop Num $j" + + for i in `seq 1 $IMFILEINPUTFILES`; + do + mkdir rsyslog.input.dir$i + mkdir rsyslog.input.dir$i/dir$i + ./inputfilegen -m 1 > rsyslog.input.dir$i/dir$i/file.logfile + done + + ls -d rsyslog.input.* + + # Check correct amount of input files each time + let IMFILEINPUTFILESALL=$(($IMFILEINPUTFILES * $j)) + . $srcdir/diag.sh content-check-with-count "HEADER msgnum:00000000:" $IMFILEINPUTFILESALL $IMFILECHECKTIMEOUT + + # Delete all but first! + for i in `seq 1 $IMFILEINPUTFILES`; + do + # slow systems (NFS) do not reliably do rm -r (unfortunately...) + rm -rf rsyslog.input.dir$i/dir$i/file.logfile + ./msleep 100 + rm -rf rsyslog.input.dir$i/dir$i + ./msleep 100 + rm -vrf rsyslog.input.dir$i + done + +done + +. $srcdir/diag.sh shutdown-when-empty # shut down rsyslogd when done processing messages +. $srcdir/diag.sh wait-shutdown # we need to wait until rsyslogd is finished! +#echo rsyslog.debug: +#cat rsyslog.debug +. $srcdir/diag.sh exit diff --git a/tests/imfile-wildcards-dirs.sh b/tests/imfile-wildcards-dirs.sh index e86d34c9f..d1cea30c6 100755 --- a/tests/imfile-wildcards-dirs.sh +++ b/tests/imfile-wildcards-dirs.sh @@ -2,7 +2,7 @@ # This is part of the rsyslog testbench, licensed under GPLv3 export IMFILEINPUTFILES="10" echo [imfile-wildcards-dirs.sh] -. $srcdir/diag.sh check-inotify-only +. $srcdir/diag.sh check-inotify . $srcdir/diag.sh init # generate input files first. Note that rsyslog processes it as # soon as it start up (so the file should exist at that point). diff --git a/tests/imfile-wildcards.sh b/tests/imfile-wildcards.sh index 9eb3602f3..fd81583fc 100755 --- a/tests/imfile-wildcards.sh +++ b/tests/imfile-wildcards.sh @@ -10,8 +10,36 @@ echo [imfile-wildcards.sh] imfilebefore="rsyslog.input.1.log" ./inputfilegen -m 1 > $imfilebefore +. $srcdir/diag.sh generate-conf +. $srcdir/diag.sh add-conf ' +# comment out if you need more debug info: + global( debug.whitelist="on" + debug.files=["imfile.c"]) + +module(load="../plugins/imfile/.libs/imfile" + mode="inotify" normalizePath="off") + +input(type="imfile" File="./rsyslog.input.*.log" Tag="file:" + Severity="error" Facility="local7" addMetadata="on") + +input(type="imfile" File="/does/not/exist/*.log" Tag="file:" + Severity="error" Facility="local7" addMetadata="on") + +template(name="outfmt" type="list") { + constant(value="HEADER ") + property(name="msg" format="json") + constant(value=", filename: ") + property(name="$!metadata!filename") + constant(value=", fileoffset: ") + property(name="$!metadata!fileoffset") + constant(value="\n") +} + +if $msg contains "msgnum:" then + action( type="omfile" file="rsyslog.out.log" template="outfmt") +' # Start rsyslog now before adding more files -. $srcdir/diag.sh startup imfile-wildcards.conf +. $srcdir/diag.sh startup for i in `seq 2 $IMFILEINPUTFILES`; do @@ -38,7 +66,7 @@ HEADER msgnum:00000000:, filename: ./rsyslog.input.9.log, fileoffset: 0 HEADER msgnum:00000000:, filename: ./rsyslog.input.10.log, fileoffset: 0 HEADER msgnum:00000000:, filename: ./rsyslog.input.11.log, fileoffset: 0 HEADER msgnum:00000001:, filename: ./rsyslog.input.11.log, fileoffset: 17 -HEADER msgnum:00000002:, filename: ./rsyslog.input.11.log, fileoffset: 34\n' | cmp -b rsyslog.out.log +HEADER msgnum:00000002:, filename: ./rsyslog.input.11.log, fileoffset: 34\n' | cmp - rsyslog.out.log if [ ! $? -eq 0 ]; then echo "invalid output generated, rsyslog.out.log is:" cat rsyslog.out.log diff --git a/tests/testsuites/imfile-wildcards-dirs-multi.conf b/tests/testsuites/imfile-wildcards-dirs-multi.conf index 5be403a43..ff93a901c 100644 --- a/tests/testsuites/imfile-wildcards-dirs-multi.conf +++ b/tests/testsuites/imfile-wildcards-dirs-multi.conf @@ -2,10 +2,12 @@ $IncludeConfig diag-common.conf $WorkDirectory test-spool /* Filter out busy debug output, comment out if needed */ +/* global( debug.whitelist="off" debug.files=["rainerscript.c", "ratelimit.c", "ruleset.c", "main Q", "msg.c", "../action.c"] ) +*/ module( load="../plugins/imfile/.libs/imfile" mode="inotify" diff --git a/tests/testsuites/imfile-wildcards.conf b/tests/testsuites/imfile-wildcards.conf deleted file mode 100644 index 0c9c4b34e..000000000 --- a/tests/testsuites/imfile-wildcards.conf +++ /dev/null @@ -1,39 +0,0 @@ -$IncludeConfig diag-common.conf -$WorkDirectory test-spool - -module( load="../plugins/imfile/.libs/imfile" - mode="inotify" - PollingInterval="1") - -input(type="imfile" - File="./rsyslog.input.*.log" - Tag="file:" - Severity="error" - Facility="local7" - addMetadata="on" -) -input(type="imfile" - File="/does/not/exist/*.log" - Tag="file:" - Severity="error" - Facility="local7" - addMetadata="on" -) - - -template(name="outfmt" type="list") { - constant(value="HEADER ") - property(name="msg" format="json") - constant(value=", filename: ") - property(name="$!metadata!filename") - constant(value=", fileoffset: ") - property(name="$!metadata!fileoffset") - constant(value="\n") -} - -if $msg contains "msgnum:" then - action( - type="omfile" - file="rsyslog.out.log" - template="outfmt" - ) -- 2.16.4
Locations
Projects
Search
Status Monitor
Help
OpenBuildService.org
Documentation
API Documentation
Code of Conduct
Contact
Support
@OBShq
Terms
openSUSE Build Service is sponsored by
The Open Build Service is an
openSUSE project
.
Sign Up
Log In
Places
Places
All Projects
Status Monitor
