grep'ing CoBOL source
in [Cobol]
|
From: Ken on 12 Aug 2010 21:13 Gentle CLC'ers, I got a couple of puzzles to work on, one technical and the other one business/career like, and I'm pretty sure at least Richard will sign on to the technical and everybody can chime in on the career/tactical one. Goes like this... My shop (about 20-25 developers, and another 20-25 business analysts, testers, managers, etc.) bought a CoBOL Analyzer tool several months ago and is trying to make it work, evaluate its effectiveness, decide on whether to sign on for upgrades and maintenance, etc. etc. I'm not sure, but I think they paid somewhere in the 5-figure range for it. Sorry, no product names will be given to protect both the guilty, and the innocent (that's ME :-). Anyway, I kinda wanted to test it out myself, but I never got an install, The Fates decided that others would be the Chosen, so I just let it go... They tried it out on the biggest single program in our system, one I've made a lot of requested changes to, about 15K LOC (without the DB2 DCLGEN and COPY expansions), and I confessed I chuckled a bit when it blew up because nested PERFORMs it was stacking to prepare a HIPO chart to wrap around the building walls blew a gasket. Yes, yes, I was gratified to feel that even at my tender gray-haired age, the PERFORM push-down stack in my brain during my visual code analysis apparently exceeds that of a 5-figure tool. No matter. It piqued my interest when I got assigned my next task, the "sticky field" task. The Manager asked of Me: Ken, Suppress the current online system's behavior of clearing the case-number field when navigating online between subsystems. The User wants the value _retained_. It's too much trouble to re-enter. It's already retained within subsystems, but is lost when moving from subsystem to subsystem (subsystem == 1st level submenu item from the top level main menu.) The Maintenance Guru immediately turned me on to a quick fix in screen MAINMENU that cleared the field in the COMMAREA (MOVE SPACES TO...) before invoking SUBMENU with another transaction. OH! Easy enough. "All ya' gotta do is disable that one line", the Business Analyst chortled :-). Maintenance Guru says, "Not so fast... Submenu 10 doesn't seem to work with that change alone. Ken, you better check out why not..." Yes, sir, Guru, and I found that SUBMENU also had a MOVE SPACES TO <sticky-field> statement as well. I heard a little voice say, "All ya' gotta do is disable that line and the first line too." :-) Hmmmm.... could _other_ "clearing statements" be lurking elsewhere, in other screens, ready to trip me up?? So I took up The Quest... Went home, went on my computer where I had installed MKS Toolkit (Unix like utilities for WinDoze/DOS), for which I have multiple paid-for versions/licenses, and copied the .exe's onto a CD-ROM. Probably all I need is grep (global regular expression print), sed (stream editor), sh (shell), and echo (display). Went to work the next day, and loaded the CD into the READ-only CD-ROM drive (No writes to external media permitted!!) drive of my work machine, went into Accessories, and opened up a DOS command-line box. Then executed a tiny autoexec.bat file on my CDE to set four or five environment variables that were needed (including PATH). They would apply only to the life of the DOS box, then die. Then I downloaded via terminal emulator FTP about 5000 COPYLIBs and 2000 program sources from the mainframe DVLP partitioned data set (PDS) onto my C: drive. No security violation, nothing squawked. Gee, I must be authorized to do this... :-). At this point, caution got the best of me, and I went to the System Admin... Me: "Hey, Scott - ya' know how we often download a program or two from the mainframe so we can cut and paste snippets into MS-Word with highlighted text and the like to conduct code reviews of our changes?" Scott: "Yeah." Me: "Well, any prohibition against using wildcarding to download the whole shebang?" Scott: "If you're not taking it out of the building, No." Me: "And Scott, as Admin, you're the only one with authority to install software on the PC's, right?" Scott: "Yep, only a guy with Admin privileges can do that, and that's me. You're not Admin. You don't even have the capability to do it." Me: "That's reassuring <smile>. So if I execute .exe's off a CD-ROM in a DOS box, that's not an Install, right? I'm not writing any executables to the C: drive, and I'm not changing the Registry." Scott: "That's not an Install. You're OK." I smiled. :-). Went back to my machine, and breathlessly entered... sh <enter> and promptly saw the friendly $ shell prompt. I changed working directories to position me over the 5000 COPYLIBs that contain procedure division code for the screens... Executed some grep one-liners until I put the debugged results into a shell script (used temp files rather than piping to help debug intermediate results.) echo "grep'ing TO XFER-BLK-BOX-ACCOUNT..." grep "TO.*XFER-BLK-BOX-ACCOUNT" *.CPY >tmpb1.txt echo "grep'ing TO XFER-BLK-BOX-ACCT-NUM..." grep "TO.*XFER-BLK-BOX-ACCT-NUM" *.CPY >tmpb2.txt cat tmpb1.txt tmpb2.txt >blkboxto.txt echo "blkboxto.txt contains all TO BLK-BOX-ACCOUNT and TO-BLK-BOX-ACCT- NUM stmnts" sed -n /ZERO/p <blkboxto.txt >blkboxzerosto.txt echo "blkboxzerosto.txt contains all ZEROS/ZEROES TO BLK-BOX-ACCT-NUM stmnts..." sed -n /SPACES/p <blkboxto.txt >blkboxspacesto.txt echo "blkboxspacesto.txt contains all SPACES TO BLK-BOX-ACCOUNT stmnts..." sed -E /\(ZERO\)/d <blkboxto.txt | sed -E /\(SPACES\)/d >bklboxotherto.txt echo "blkboxotherto.txt contains all <<other-than-spaces-or-zeros>> TO stmnts..." echo "Done!" It executed and scanned the 5000 COPYLIBs in less than two minutes :-). Result: I had files of occurrences of three kinds of statements - o One file of about 5 or 6 MOVE SPACES TO <sticky-field> (including MAINMENU) o One file of about 8 or 10 MOVE ZEROES TO <sticky-field-redefined-as- PIC9's> o One file of about 40 or 50 MOVE <other-data-element> to <sticky- field> Gleefully entered the conference room with Maintenance Guru, Team Lead, Business Analyst, and Functional Analyst.. "Yeah, all we gotta do is disable the clearing statement in the main menu... - and in this Option 10 submenu... - and in these other two or three submenus... - and check out these other 8 or 10 screens farther down the subsystem screen navigation path to see if its safe to disable there too, and then test the screen navigation paths around all of these we've changed. The Maintenance Guru grinned, the Team Lead looked a little quizzical, and the analysts looked a little mystified that something so simple could require modifying more than a couple of screens, but they were otherwise agreeable. But I was not finished... "OH! And one more thing, just to make sure... I know you want the value _retained_, but when there is a screen whose intent is to _change_ the value, then _that_ new value becomes the sticky value, and it's OK the original one is lost, right? Here's a list of about 50 screens out of the 600 or so that do that. I think I got them all, but I might have missed a couple." Everybody agreed that indeed this was what was wanted, so we adjourned amicably. Afterward, Team Lead took me aside to ask, "Ken, why did you present that list of screens where the case-number is reset to nonzero?" Me: "Oh, just so that there were plenty of examples for the Analysts to know what I was talking about." He must have been wondering why two or three examples were not enough for me, and not knowing how I did it, he must have imagined me single- stepping ISPF 3.14 through 5000 COPYLIBs representing 600 screens, searching for MOVE TO sticky-field statements. I laughed to myself. :-) If you're still with me, here's the two questions... 1. Technical. OK, I left some stuff out above. I also grep'd for INITIALIZE <sticky-field>, but that didn't turn up anything. The tougher situation is when <sticky-field> appears on a source line by itself, or with a comma, or a period, or the like, and the CoBOL MOVE statement wraps around multiple source lines. grep and sed work on one line at a time. I'm pretty sure that I could have programmed something in the awk utility to cover line wraparound, but didn't want to take the time. Instead I grep'd for this to identify the "standalone" lines... grep -E "^...... *,? *XFER-BLK-BOX-ACCOUNT *[,.]* *[0-9]*$" *.CPY >xferd07alone.txt (The beginning and end of this regular expression string is to account for line numbers at the beginning and end of the record.) There weren't too many of those, 20 or 30, so I just went into the COPYLIB source for each of them to see how they were used. Here I did severalt that were being INITIALIZEd by an INITIALIZE verb a few source lines above, and I found a couple where sticky-field was one of several destinations for MOVE SPACES. I added these to my lists. So, question - will the grep above catch 99% of the "isolated" data- element lines? OK, I guess it wouldn't catch the following line 123456 , XFER-BLK-BOX-ACCOUNT END-IF 12345678 but usually if we have something like... IF something-or-other MOVE SPACES TO element-1, element-2, element-3, element-4 END-IF .... we have the END-IF on a line by itself, right? And not on the same line as element-4? OK, you know I'm not talking what is syntactically correct, but rather what is the preponderance of coding style across the industry. Anyway, I'm just looking for a "succint" regular expression to grep and find nearly all occurrences of a data-element appearing on a line by itself. Regular Expression syntax is still a bit of a challenge for me, and I usually approach it by starting with the most loosely defined one that matches too much stuff and then I add more and more to the expression to zero in on the "hits" that I am really looking for. 2. Career/Political Question - - What do I do now? I mentioned this tool capability to one low-level manager, and he was kinda interested, but he is not yet following up with me... hmmm.... might be conflicting with a more expensive tool that doesn't do as much, but just as easily Top Manager maybe wants to kill that one, so maybe it's OK, maybe not. Can't get a good read. - Uhhhhh --- OK, I've got multiple licenses, so _probably_ I've got that angle covered as far as I'm not using unlicensed software, but maybe there's some fine print somewhere that says I can _Install_ it on a machine, but I can't make "copies" of it onto a CD... :-) --- but I'm not gonna share it willy-nilly across the floor, I'm not into that, so probably I'm low-profile enough to be Ok...I'm either going to be the only user, or the shop will have to pony up for full install licenses (they would want that for the support, anyway.) - Cost is only about $500 a seat - do I promote this capability and request 5-6 copies for the managers and a few key techs like me? Or do I "fly below the radar" best I can, quietly execute this tool myself and myself only, be the "best" maintenance grunt :-) onsite as far as finding system impact of cross-referenced changes like this, and feather my own job-security nest that way? - OK, OK, I could ask He Who Signs My Timesheet, but his fortunes may be tied to Expensive-Tool (there are clues that this is at least somewhat true), and do I really need his permission to do what I'm doing? From what I can tell I haven't violated any policy (haven't taken company work product offsite, haven't installed unauthorized software), all I've done amounts to bringing in a Cross pen to write with instead of a Bic pen? (A bit of an overstatement, but you get the point.) - Or do I call an end to what has been an interesting experiment, and go back to using ISPF 3.14 over and over again, for after all, I am paid by the hour, and the Customer has been satisfied so far... Your humbl' CLC occasional contributor, Ken
From: Anonymous on 13 Aug 2010 09:08 In article <098a698a-a3da-4e55-b6b5-12a58ee646f9(a)d8g2000yqf.googlegroups.com>, Ken <klshafer(a)att.net> wrote: >Gentle CLC'ers, Oh... ruling me out from the start, eh? [snip] >My shop (about 20-25 developers, and another 20-25 business analysts, >testers, managers, etc.) bought a CoBOL Analyzer tool several months >ago and is trying to make it work, evaluate its effectiveness, decide >on whether to sign on for upgrades and maintenance, etc. etc. I'm not >sure, but I think they paid somewhere in the 5-figure range for it. Ahhhh, the old 'buy it first and let someone else see if it works' routine, that way the person authorising the purchase can always said 'Well sure *I* bought it but (name of subordinate) couldn't make it work so It Isn't My Fault.' (never mind that the purchase can be written off/depreciated as can any other business expense... something Cost Money so fingers must be pointed) [snip] >"All ya' gotta do is disable that one line", the Business Analyst >chortled :-). > >Maintenance Guru says, "Not so fast... Submenu 10 doesn't seem to work >with that change alone. Ken, you better check out why not..." This Guru fellow seems *very* secure in his position; responding to an 'All ya gotta do' in that way seems to indicate such. > >Yes, sir, Guru, and I found that SUBMENU also had a MOVE SPACES TO ><sticky-field> statement as well. I heard a little voice say, "All ya' >gotta do is disable that line and the first line too." :-) > >Hmmmm.... could _other_ "clearing statements" be lurking elsewhere, in >other screens, ready to trip me up?? At this point one turns it into a Matter of Simple Mathematics... haw haw haw! 'Gosh, Mr Corner-Office Idiot, I have a numeric sequence that begins with zero, one and two; all I gotta do is find out whether it continues in unitary digit incrementals (following the sequence of intergers) or does it shift into an exponential series (computers do stuff like that, y'know... they go from 2 to 4 to 8 to 16 to 32 really easily and you never know when you're going to encouter *that*)... ' (and at this point eyes should begin to glaze and you Move In For The Kill, the prize of the hunt, in this instance, being enough time to do what's needed without stinkwads stopping your cubicle every 37.5 minutes saying 'So how far have you gotten on this?', and you should muster enthusiasm and glee for the next) '... but since we all know that Things Are Different Here maybe it's something *really* fascinating, like a Fibonacci sequence of 0,1,1,2,3,5,8,13,21... oh, *you* know the rest... and they're found all over the place, like the distribution of seeds in a sunflower spiral, the shell of the nautilus (the cephalopod, not the submarine in the Jules Verne story... or that exercise-device they used to advertise on television... I wonder whatever happened to those) and the surface-cells in pine cones... just fascinating, the 'Golden Ratio', or 'Phi'. of 1:1.618 just kind of falls right out of it...' .... and at this point everyone in the room should be willing to cut you a five-week task order just to get you out so they can get back to talking about golf or sports scores. > >So I took up The Quest... [snip] >Gleefully entered the conference room with Maintenance Guru, Team >Lead, Business Analyst, and Functional Analyst.. > >"Yeah, all we gotta do is disable the clearing statement in the main >menu... > - and in this Option 10 submenu... > - and in these other two or three submenus... > - and check out these other 8 or 10 screens farther down the >subsystem screen navigation path to see if its safe to disable there >too, and then test the screen navigation paths around all of these >we've changed. > >The Maintenance Guru grinned, the Team Lead looked a little quizzical, >and the analysts looked a little mystified that something so simple >could require modifying more than a couple of screens, but they were >otherwise agreeable. > >But I was not finished... > >"OH! And one more thing, just to make sure... I know you want the >value _retained_, but when there is a screen whose intent is to >_change_ the value, then _that_ new value becomes the sticky value, >and it's OK the original one is lost, right? Here's a list of about 50 >screens out of the 600 or so that do that. I think I got them all, but >I might have missed a couple." > >Everybody agreed that indeed this was what was wanted, so we adjourned >amicably. Another classic... 'we always want it done (this way) except when we don't'. > >Afterward, Team Lead took me aside to ask, "Ken, why did you present >that list of screens where the case-number is reset to nonzero?" > >Me: "Oh, just so that there were plenty of examples for the Analysts >to know what I was talking about." I would say this was a missed chance to Establish Precedence. Not that I am an example of anything special... but when I am asked 'Why did you do this?' and the immediate response is not 'Because (person of sufficient authority) told/asked me to' I reply 'I thought it was a Good Idea at the time; it allowed for...' and in this case the follow-through is 'the Analysts to have plenty of examples'. In other cases it may be something like 'I thought it was a Good Idea at the time; it demonstrated the pervalence of the situation so that time could be saved avoiding other facile (fancy-talk for 'all ya gotta do') attempts at resolution' or 'I thought it was a Good Idea at the time; it showed the possible cost-reductions would be balanced against the maintenance and testing costs'... .... but the point is *not* to give an actual reason, the point is to get 'I thought it was a Good Idea at the time' to be an acceptable response from you. [snip] >1. Technical. OK, I left some stuff out above. I also grep'd for >INITIALIZE <sticky-field>, but that didn't turn up anything. The >tougher situation is when <sticky-field> appears on a source line by >itself, or with a comma, or a period, or the like, and the CoBOL MOVE >statement wraps around multiple source lines. I was wondering when this might appear. Since I'm unfamiliar with the 'all ya gotta do' to make grep and sed and such work I would have used something like: //SEARCH EXEC PGM=ISRSUPC, // PARM=(SRCHCMP, // 'ANYC') //NEWDD DD DSN=YOUR.PDS.NAME, // DISP=SHR //OUTDD DD SYSOUT=* //SYSIN DD * SRCHFOR ' STICKY-FIELD' CMPCOLM 12:72 /* This will process all the members of a PDS (and search only between columns 12 and 72, saving Valuable CPU Time). Note the space after the first ' in the SRCHFOR card; this is used to eliminate hits on WS-STICKY-FIELD or TEST-STICKY-FIELD. (if such hits are wanted then remove the space and snug the ' up to the S) [snip] >2. Career/Political Question - I'm really no good at all at addressing such matters; I manipulated my brag-sheet to show two years' worth of experience as an employee and have been 'topped out' as a consultant/contractor/hired gun ever since. DD
From: Fred Mobach on 13 Aug 2010 11:14 Ken wrote: <<snip technical details>> > Executed some grep one-liners until I put the debugged results into a > shell script (used temp files rather than piping to help debug > intermediate results.) > > echo "grep'ing TO XFER-BLK-BOX-ACCOUNT..." > grep "TO.*XFER-BLK-BOX-ACCOUNT" *.CPY >tmpb1.txt > echo "grep'ing TO XFER-BLK-BOX-ACCT-NUM..." > grep "TO.*XFER-BLK-BOX-ACCT-NUM" *.CPY >tmpb2.txt > cat tmpb1.txt tmpb2.txt >blkboxto.txt > echo "blkboxto.txt contains all TO BLK-BOX-ACCOUNT and > TO-BLK-BOX-ACCT- NUM stmnts" > sed -n /ZERO/p <blkboxto.txt >blkboxzerosto.txt > echo "blkboxzerosto.txt contains all ZEROS/ZEROES TO BLK-BOX-ACCT-NUM > stmnts..." > sed -n /SPACES/p <blkboxto.txt >blkboxspacesto.txt > echo "blkboxspacesto.txt contains all SPACES TO BLK-BOX-ACCOUNT > stmnts..." > sed -E /\(ZERO\)/d <blkboxto.txt | sed -E /\(SPACES\)/d >>bklboxotherto.txt > echo "blkboxotherto.txt contains all <<other-than-spaces-or-zeros>> TO > stmnts..." > echo "Done!" > > Instead I grep'd for this to identify the "standalone" lines... > > grep -E "^...... *,? *XFER-BLK-BOX-ACCOUNT *[,.]* *[0-9]*$" *.CPY >>xferd07alone.txt > (The beginning and end of this regular expression string is to account > for line numbers at the beginning and end of the record.) > > There weren't too many of those, 20 or 30, so I just went into the > COPYLIB source for each of them to see how they were used. Here I did > severalt that were being INITIALIZEd by an INITIALIZE verb a few > source lines above, and I found a couple where sticky-field was one of > several destinations for MOVE SPACES. I added these to my lists. > > So, question - will the grep above catch 99% of the "isolated" data- > element lines? OK, I guess it wouldn't catch the following line > 123456 , XFER-BLK-BOX-ACCOUNT > END-IF 12345678 > but usually if we have something like... > IF something-or-other > MOVE SPACES TO element-1, > element-2, > element-3, > element-4 > END-IF > ... we have the END-IF on a line by itself, right? And not on the same > line as element-4? OK, you know I'm not talking what is syntactically > correct, but rather what is the preponderance of coding style across > the industry. That would be a difficult answer to your question. But I might try. You can concatenate a lot of lines with sed (before your grep), where the last line of a serie should have a single dot for termination of a COBOL statement. In the grep statement each line terminator like <LF> will be changed to a space when not preceded by a dot. In that way grepping on source and destination fields will be more simple with a regular expression. BTW, since some recent sed versions the --in-place option can be used to avoid temporary files. > 2. Career/Political Question - > - What do I do now? I mentioned this tool capability to one low-level > manager, and he was kinda interested, but he is not yet following up > with me... hmmm.... might be conflicting with a more expensive tool > that doesn't do as much, but just as easily Top Manager maybe wants to > kill that one, so maybe it's OK, maybe not. Can't get a good read. > - Uhhhhh --- OK, I've got multiple licenses, so _probably_ I've got > that angle covered as far as I'm not using unlicensed software, but > maybe there's some fine print somewhere that says I can _Install_ it > on a machine, but I can't make "copies" of it onto a CD... :-) I don't have read the license of your software as we don't have MS-Windows software on our networks, but both grep and sed are available in multiple Free Software servions, as in you have the right to study, modify and redistribute it like you want. But to give rest to others problems, you might select an older box, install a Linux distribution, a BSD version or even an OpenSolaris version on it and run FTP, grep and sed without retributions. Good luck with your 5 figure savings. :-) -- Fred Mobach - fred(a)mobach.nl website : https://fred.mobach.nl .... In God we trust .... .. The rest we monitor ..
From: Ira Baxter on 13 Aug 2010 11:39 "Ken" <klshafer(a)att.net> wrote in message news:098a698a-a3da-4e55-b6b5-12a58ee646f9(a)d8g2000yqf.googlegroups.com... > > My shop (about 20-25 developers, and another 20-25 business analysts, > testers, managers, etc.) > I changed working directories to position me over the 5000 COPYLIBs > that contain procedure division code for the screens... > > Executed some grep one-liners until I put the debugged results into a > shell script (used temp files rather than piping to help debug > intermediate results.) > > echo "grep'ing TO XFER-BLK-BOX-ACCOUNT..." > grep "TO.*XFER-BLK-BOX-ACCOUNT" *.CPY >tmpb1.txt > echo "grep'ing TO XFER-BLK-BOX-ACCT-NUM..." > grep "TO.*XFER-BLK-BOX-ACCT-NUM" *.CPY >tmpb2.txt > cat tmpb1.txt tmpb2.txt >blkboxto.txt > [...more greps...] > sed -n /ZERO/p <blkboxto.txt >blkboxzerosto.txt [finding essentially MOVE ZERO/SPACE to XFER-BLK-... targets] >It executed and scanned the 5000 COPYLIBs in less than two >minutes :-). Yes, grep was invented so people could fish around in big piles of code quickly. And that's really useful, as Ken has shown. The good news is that grep is pretty fast. The not so good news is that is only pretty fast; he had to run it bunch of times. Note he was piping the results to temp files to record what he did, so he could decide if the answer was good. He didn't say it, but presumably he had to go open the files that appeared interesting to verify his hits. There are alternatives to grep which are faster and more convenient. They are called "source code search engines". OpenGrok (hub.opensolaris.org/bin/view/Project+opengrok) is one of them (opensource). It indexes and will thus search for identifiers very quickly; it does this by knowing enough about the various langauges you might work with to pick out identifiers. It also searches for abitrary strings with regexes at grep-like speeds, and will show you lists of hits and offer to show you the code directly. Another is our Source Code Search Engine (commercial) (www.semanticdesigns.com/Products/SearchEngine) The SCSE knows the lexical structure (language tokens: keywords, identifiers, numbers, comments, strings, whitespace) of a variety of languages [e.g., Java, C#, C++, ) but especially including COBOL (and specifically AS400 and IBM Enterprise COBOL). SCSE indexes *everything*, so it has lightning fast searches for identifiers and combinations of tokens for complex queries. Ken's search could have been written to the SCSE as: 'MOVE' ('ZERO'|'SPACES') 'TO' I=XFER-BLK-BOX-ACC* (In fact for this query you don't really need the 'MOVE' keyword). You don't need to say anything about the whitespace, because SCSE knows that whitespace isn't interesting (by definition, in *any* language, for each langauge's defintiion of whitespace). You don't need to worry about line breaks or formatting, because they are whitespace, too. Response times are seconds across millions of lines of code, thousands of files. Instead of temp files, he would have seen a GUI list of individual lines across *all* files containing his hits; a single click from there would pull up the the source file with the hit highlighted. Such tools really are convenient ways to get "good information, really fast". I think the other tool Ken mentioned is one that brings up "great information (maybe), much more slowly". -- IDB
|
Pages: 1 Prev: Cobol is Dead Next: Commas, et al (qs: amount in words in cobol any built in function |