At companies I've worked at with mainframes the teams running them were typically very well tenured on both the systems and software running on them.

They weren't the types to go looking on StackOverflow or Youtube for answers.

They would also help train new joiners, and IBM offers good training too.

The kinds of systems running on mainframes don't lend themselves to copy/paste from StackOverflow type programming. It's a lot of credit-card and banking transactions where you want experienced people writing and helping on it.

Secondly, even if IBM is selling more than before, that's still an incredibly tiny fraction of the computing scene. Companies don't tend to have more than one (or possibly 2, with one for DR mainframes). So, that 80-90% of the Fortune 500 is really just 1000 mainframes total. Generously, if every Fortune 500 had a pair in each geography (roughly EU, Americas, Asia for big corps), that ends up being 6, so 3000 globally, in production. There are also some spread around academia, government labs, etc. But the numbers are dwarfed by standard x86 machines. eg. at one bank we had "the mainframe" which was actually one hot and one DR, and then 40k windows servers and 40k linux servers. Similarly, there were thousands of engineers in technology, but the mainframe team with single digits.

Given the wealth of information out there, even if there was "a lot" of information for some value of lots, it's dwarfed by the incredible amount of standard tech info.

StackOverflow, I feel, inherently ties into the "Release early. Release often." philosophy: if you ship broken code , oh well , there'll be a release soon enough to fix it. This can not be borne in a mainframe environment. This is not to say there aren't bugs because of course there are but the expectation is completely different. This kind of "by your bootstraps" education is no bueno. You will get lots of training and excellent documentation. We used to measure the amount of documentation IBM ships with the mainframe not in pages but in yards it takes on the shelves.

StackOverflow in itself is the answer to the question "who you gonna call" -- because you might be alone, or you might be working on a small team where no one has better expertise, and usually you are working with open source and so no one is obligated to pick up the phone at the software "manufacturer". This scenario does not play in a mainframe environment: you are going to be part of a larger team, full of literal graybeards and when push comes to shove then you do have a contact number: your employer is paying absolutely ridiculous amounts of money to the software company and they will have a contact person who will pick up the phone.

There is an emulator for running mainframe code called Hercules. It comes with no software, so you'll need to acquire compilers & database apps separately.

The equivalent from IBM is called "IBM Z Development and Test Environment" and the license for that costs $5,540 per user per year. There was (maybe still is?) a "learners edition" that you have to contact them to "see if you qualify" and it is a far more reasonable $120/year (there are no links on IBM Marketplace to even try to order it). This includes several compilers and 3 databases: CICS, DB2 & IMS.

IBM has a very different model for supporting developers than Microsoft does with MSDN.

Links:

https://en.wikipedia.org/wiki/Hercules_(emulator)

http://www.hercules-390.org/

https://www.ibm.com/products/z-development-test-environment/...

https://ibm.github.io/zdt-learners-edition-about/

https://old.reddit.com/r/mainframe/

Ballmer being excited: https://www.youtube.com/watch?v=wNy2zBG79zU

A shop I worked in ages ago had a mainframe on site supporting lots of batched transactions over the course of a 24 hour period, along with the daily operations for a 35 facility logistics company, spread throughout the southeastern United States. They had a staff of 8 mainframers that ran the entire operation. All of the PCs onsite (my area of responsibility) were basically 3270 terminal clients for z/OS.

Sometime later, a connection on LinkedIn pointed out the following resources:

https://www.openmainframeproject.org/

https://github.com/openmainframeproject

https://github.com/openmainframeproject/cobol-programming-co...

Also, mainframes are really good at high throughput transactional jobs. That's why you see them in banks, transportation, insurance, etc. Big Tech™ doesn't see it as "cool" and are too focused on the Next Big Thing™, so there's not a lot of attention there. Sometimes, boring just gets the job done.

I know a few mainframe folks. I don't think there's a single answer, but there are definitely a few inhibiting factors.

* The documentation for mainframe stuff is incredibly comprehensive compared to what most people are used to, so there's less need for something like Sewer Overflow.

* The code people are working on is likely to be proprietary, so no GitHub.

* Mainframes are often used in security-conscious environments where "openness" is not a core value and might even be considered negative.

* People working in that area might like being part of a relatively closed community (lower supply = better pay) so they don't advertise.

And then there's just critical mass. Since there aren't already a lot of mainframe programmers Out There, and mainframe programmers' work has few technical "touch points" with the rest of the world (e.g. different languages, libraries), it almost makes more sense to flip the question around.

Why would there be more info on the web about IBM mainframe programming?

There isn't a lot on Stack Overflow and the rest of the web because it isn't needed.

Mainframe programming is not like web programming. There are not multiple frameworks and languages competing for developer attention and rising and falling in popularity every few years. It is much more "there is one way to do it, it's the same way we've done it for 15 years" and that is what you do. There are very few new questions. All the common questions are covered in training and the more uncommon ones are covered in the documentation. If there isn't already a standard way to do what you want to do, you are probably doing it wrong.

> Most of the results are on IBM's own website, and even that contains only high-level details.

As other people have stated in this thread, IBM Redbooks are what you need and they span the whole range from high level overflow to low level.

The ABC's of System Programming is probably where you want to start, there are 13 volumes I think.

https://colinpaice.blog/2020/06/22/whats-in-abcs-of-z-os-sys...

You just have to know what to look for to find low level stuff - for an example, here is the reference for IBM's High Level Assembler Language (HLASM).

https://www-40.ibm.com/servers/resourcelink/svc00100.nsf/pag...

> And if there are, then why is there so little info for them on the web? Is the knowledge proprietary and secret?

IMO it is just a cultural thing. A lot of mainframe developers and especially the experts who have deep knowledge have been programming for a long time. Like, started on punch cards while they were getting a math degree.

The IBM culture has always been comprehensive documentation available from the source and predates the internet by a few decades, so the kind of public stack overflow culture we are used to never developed over there.

The book Hackers: Heroes of the Computer Revolution by Steven Levy kind of talks about this, the difference between the mainframe guys and the mini computer guys. Its a good read.

Mainframe people are descendants of the "high priests" that worked on computers before the invention of the mini computer and the culture reflects that to this day.

I suspect a lot of mainframe development today is Java EE centered, for which there is an abundance of stuff on the web.

During my vocational training, I did a bit of work with mainframes and found a forum back then I liked, but my work (and the forum) was mostly about sysadmin-stuff, and the little coding I did was in Perl.

There is a sizable library of RedBooks on IBM's web site somewhere, but again, the stuff I was interested in was about system administration.

A lot of the culture in the mainframe world developed a long time ago, and in relative isolation from the rest of the IT world, so I suspect by the time Stack Overflow became popular, their communities had already found other ways and places to mingle and pass knowledge on and around.

With hardware, software, and support contract being so expensive, and corporate culture being what it is, I also suspect the threshold to call up the the vendor for support instead of Stack Overflow is much lower than in most other areas.

(Most of this is speculation on my part, I only spend a few months in this fascinating corner of IT. If I'm wrong, I'd love to hear to from someone with actual experience.)

In the late 1990s, when I joined a bank that used IBM mainframes, there was a group of 30ish new CS graduates who would spend many months being trained internally and by IBM to learn to program and operate those. As I understand it, that's still the case today.

The people who operate these mainframes have tremendous access to internal IBM resources. I remember having to stifle my laughter as, during a meeting about a major outage, a seasoned programmer explained that he "took a dump and sent it to IBM" and was expecting to hear back any time.

The channels that exist for this knowledge are expansive and just aren't on the internet, though they do occasionally spill over there.

Much of the expertise on the platform has been slowly internalized by IBM over the years. The experts on the platform work for them.

Example. I used to work at a massive organization that did have some core applications on mainframes. The guys who maintained it retired and they couldn't find anyone to fill the position. So they contracted with IBM to maintain the system. Guess who showed up on site wearing IBM polos? The guys who just retired. Now doing the same job but making even more while doing it.

I went through a mainframe cobol bootcamp at a company in the 90's and worked on their code for a few years. The thing to know today is most all the jobs (in the US) were offshored, very few of those jobs are left in the US today.

The other thing to know is; it's really hard to learn if you don't have access to one (obviously). They are from an era when computing was not ubiquitous, there is a whole parallel universe of computing in mainframes where things have different names and conventions, while the basic concepts are the same. It can be very hard to translate.

For me, my time on mainframes was enlightening and interesting, but a major career dead end. I had to go back to entry level in a different stack to get out of it. Glad I did, those jobs are all dead end, miserable, cost centers. Every company who has mainframe applications would love to get off them, but they can't because the cost is too high.

A lot of the documentation is printed in manuals/books written before the 90s. I have inherited a whole closet worth of COBOL programs (on both magnetic tape and print out on continuous form paper via dot matrix) and mainframe documentation from my parents who made a career in COBOL programming. Many of the people who worked on those systems predated the internet and the concept of sharing information via the internet was not part of the culture. It's one thing I admire about today's devs- the amount of free and high value information on YouTube, Twitch, GitHub and Stack Overflow.

I encourage young software engineers to seriously look at Mainframe development as a possible career path. Currently there is an aging knowledge base that needs to be replaced. Also, that world lends itself more to continually leveling up (i.e. experience matters more) than the typical development world where technologies change so fast.

On the down side it's much harder to bootstrap an environment for doing side projects on the mainframe. The hercules emulator makes it possible locally but it's a pain [0]. However, I think IBM provides sandox's for people to mess around with and all that requires is some minimal sign up and a TN3270 terminal emulator.

[0] http://www.hercules-390.org/

You'll find mainframes in many/most large older (founded before 1990) organizations but almost nowhere else. There doesn't tend to be a lot of detailed information publicly available for proprietary enterprise products generally. This is partly because that's the way many companies prefer it (i.e. trade secrets and NDA's) and partly because they just aren't of general interest to hobbyists/hackers/smaller customers. In addition to the lack of information, mainframes also have significant physical/electrical/cost constraints which further limits interest in terms of actually owning one... which would be a driver of interest.

Compare this to your typical PC/mobile device which fits on/in a desk/lap/pocket, is practically free in comparison, has a deluge of information available everywhere and is actually useful to most people in everyday life. About the only thing a mainframe really gets you is reliability in a monolithic device which from a cost/benefit standpoint isn't worthwhile for most customers and can be approximated via other means.

I've been inside many organizations that run mainframes and if you're not on one of the mainframe teams, which tend to be tiny relative to other IT groups, it's pretty much plumbing as far as the rest of the organization is concerned... which is the point.

A few years back I did something called 'master the mainframe' where they provided a TSO login for you on an instance running in one of their labs. They also provided you with a bunch of fun exercises to learn more about the system.

I think this is the equivalent - https://ibmzxplore.influitive.com/users/sign_in

There's stuff out there, it's just not always easy to find. A lot of the IBM mainframe stuff, for example, is buried in their catalog of "Red Books" found at www.redbooks.ibm.com.

Check out:

https://www.ibm.com/z/trials

https://www.redbooks.ibm.com/search?query=%22system%20progra...

https://developer.ibm.com/?q=z%2Fos

Also search ibm.com for terms like "PL/I", "COBOL", "CICS", "RPG" and so on and you'll likely turn up more of the mainframe stuff. You can, of course, program mainframes in other languages, especially Java. But from an IBM perspective, references to the former batch of terms will generally turn up stuff about their mainframes, or their minicomputer series (mostly what's called iSeries these days, as far as I know. Formerly the AS/400, S/38, etc).

There are also various print books from 3rd party sources that can be found on Amazon. Ex:

https://www.amazon.com/IBM-COBOL-Complete-Guide-2020/dp/0655...

https://www.amazon.com/Quick-Start-Training-Application-Deve...

etc.

The reasons for this are, so far as I can tell, largely what other commenters have already posted. It's just a different culture and way of doing things, and the mainframe world just works a bit differently. And I think a lot of that is because that world is so insular now with IBM being more or less the only mainframe vendor left. I suppose Unisys or Hitachi or Fujitsu or somebody may still make a mainframe here and there, but the market is almost completely dominated by IBM. So the "mainframe way of doing things" is more or less "the IBM way of doing things" and in particular it's a very old skool version of "The IBM way" at that.

We actually published a research paper recently on COBOL communities, but I still don't have a great answer to your question.

Much of the information is locked away in developers' heads and it was difficult for us to find many resources on getting started with COBOL. In fact, most of the COBOL code on GitHub is homework assignments but almost no real-world programs that we could analyze.

Our paper which contributes a defects benchmark: https://austinhenley.com/pubs/Lee2022ICSME_COBOL.pdf

Have you seen https://www.ibmmainframer.com/?

I would also dig through Project Guttenburg https://www.gutenberg.org/ for IBM Mainframe Programming books if you're interested in the history.

https://github.com/samanL33T/Awesome-Mainframe-Hacking

https://github.com/FuzzyMainframes/Awesome-Mainframes

I worked somewhere where almost everything was run on some kind of mainframe, but it was abstracted from 90% or more of the developers, who wrote more or less normal code in normal languages. The “mainframe programming” was basically a systems level or infra type role providing this platform to everyone else. I imagine this is equivalent to a k8s team in other companies. However I’m sure there are plenty of banks and other places that are actually writing business logic with the mainframes’ toolset.

I did hear IBM has free courses for people wanting to learn mainframe. For example:

https://www.ibm.com/blogs/ibm-training/free-course-announcin...

I am sure there are others out there too.

Maybe because someone needing a fact on the mainframe realizes that almost no one on StackOverflow will help them, plus many will make fun of them for being a dinosaur?

Edit: I should add that I'd bet they DO have their own web forums, and a lot of them are probably run by IBM itself. (I'm not a mainframer, in case you're wondering)

Have you seen: https://www.ibm.com/docs/en/zos-basic-skills

I've no knowledge of IBM mainframes - but I assume they're sold as part of a large consultancy/training package.

This is an interesting question actually and it leads me to draw parallels with some IRC channels, like for example #vmware and #rhel at libera (formerly freenode).

These are what I would call Enterprise environments, IBM, VMware, RHEL. And they all have one thing in common, lack of community documentation. Because in Enterprise environments you're expected to be trained.

And in the IRC channels people mostly talk about politics, or just everyday stuff. Very rarely do they nerd out about Linux or hypervisors. Because most people in there are trained engineers and neither need help, nor are willing to give help to someone.

If you go up to a crowd of enterprise people to ask a question, you better ask a good question. The old adage "there are no stupid questions" does not apply at all. Because these are trained people who have very specific knowledge of their Enterprise systems. They don't mess around with community projects or open source for fun. They do it to achieve a goal.

I know someone who started at IBM about a year ago and he's learning quickly how to program on IBM mainframes. By other senior engineers, and trainers.

While others addressed different parts already, I just wanted to mention that the IBM (well, any Corp) marketing data should be treated like it's written to be technically/legally correct but stretched as positive as possible.

> "ninety percent of Fortune 500 companies rely on the IBM Mainframe"

This doesn't say they own them. It's also very vague given they know the exact number.

> "eighty percent of the worlds corporate data resides or originates on the mainframe."

This doesn't say it's the primary store, or is that by volume or dataset count, or what constitutes "data". It doesn't say where the chain ends either. If the user record is created on a mainframe, did everything related to the user originate on the mainframe?

Take it with appropriate amount of salt.

Look for IBM Redbooks, also the "Mainframe" has 5 different operating systems, zVM, OS2, zOS, Linux (s390x architecture), and somehow I forgot the other.

There is Code in a multitude of languages even when it is not Linux, from C++ to Java, but nowadays also nodejs and whatever you want to run in that architcture.

Edit:

Basically, you should look for "System Z" which is the actual name of the machine, and "redbooks" the name of the documentation manuals.

Example: https://www.redbooks.ibm.com/domains/zsystems

https://www.redbooks.ibm.com

has a lot off z howto books

Ah yes the yearly Mainframe thread on HN. Many years ago (10+) one of these threads convinced me that I should get into mainframe programming. But I couldn't figure out how to get in. The actual job postings all require 5+ years of experience with Mainframes and I'm starting from zero. And there aren't that many job postings, I'd say one mainframe job posting for every 1,000 or 10,000 web developer roles. So why bother?

What are some mainframe jobs (developer, sysadmin, others) and what do they pay? Are any of these jobs in SV or remote?

I cut my teeth on A/UX and SunOS so I'm of a certain age, but besides having school accounts for intro/101 courses using 3270 emulation I never really got hands on. For the same reasons that today I still wish Solaris had won, I think I would enjoy certain aspects of the mainframe environment.

I wonder if any of my modern stack skillset would be useful or not.

There aren't resources to practice it but IBM documentation is actually really really good it reminds me of MSDN in the 2000s. So if you can get access to an environment to practice the documentation is really solid. It surprises me that you complain about IBM website because you can get everything there, even detailed machine instruction reference for z/Architecture if you want.

Companies have been trying to bury the mainframe for 40+ years in search of "cheaper" alternatives. Software companies, with few exceptions, have followed suit. The early implementations of "Open Systems" software on the mainframe were generally unmitigated performance disasters. It's a bit better now, but not something you'd want to bet the bank on.

Not that it proves anything, but here's a S.O. post from yesterday...

https://stackoverflow.com/questions/74593657/dsnacics-stored...

Compared to other stacks/platforms, it may seem a dearth, but given the number of programmers working in that area (and as others here have pointed out, given the very different nature of mainframe programming in general), it's no so surprising that there's relatively little info that's searchable on the public internet. TBH I'm still frequently surprised by not being able to find information about a particular error I'm getting with a particular API that I would have thought was in relatively common use even in the C#/.NET world!

These are good resources:

https://mainframenation.com/mainframe/how-to-get-a-mainframe...

https://www.youtube.com/@moshix

You get physical manuals as part of your support agreement, like a bread truck full. In my experience it was something you learned by sitting next to someone who had been doing it for decades and they just call each other if they forgot something. The EBCDC chart was the most used doc.

You can finds lots of books and online resources on COBOL programming or JCL. There are also encyclopedia sized publications called IBM Redbooks (available in pdf if you Google for them) which provide tons of detail on hardware and OS level concepts you need background understanding of.

As for how you'd actually play around with writing usable software.. you'd need access to a mainframe. Basically, the insane complexity of mainframes and the walled garden nature of the ecosystem makes it extremely difficult to teach or to learn.

I worked on the z/OS network stack for a few years (tech support and later some actual development) and it was the most complex thing I've ever tried to learn in my career.

IBM historically was the most proprietary of the most proprietary. Like, if there were ANY information sharing out there in public about mainframe programming the way there is about Java/JavaScript/Node/Go/Rust or even Windows programming today, that would get you visited by an army of men in suits. Not government agents, but just about as scary: IBM lawyers.

So that cultural value of keep everything close to the vest and only learn through official channels, still persists in IBM mainframe culture today. Especially since there is little to no "hacker culture" surrounding mainframes since it's not like you can download z/OS and run it on your PC (things like Hercules notwithstanding).

There's a ton of information on IBM mainframe programming, they're the IBM redbooks. They're all here:

https://www.redbooks.ibm.com/domains/zsystems

IBM's mainframes have some of the best technical documentation for any product line, anywhere, and if you can't find what you're looking for in the redbooks, the expensive support contract that you likely got with your z/Series comes with support that's sometimes described as... aggressive at solving high priority incidents with their internal support databases.

There's also the online manuals that come with z/OS, and you can get a free account via z Xplore: https://www.ibm.com/it-infrastructure/z/education/zxplore

You can also get a Linux LPAR on a s390x machine (LinuxONE, basically a regular mainframe with the processors fused to only run Linux) as part of the free tier on IBM Cloud.

That being said, most real-world deployments have a ton of other stuff running on top of the OS, usually from Broadcom and other close software partners, and dealing with those packages usually requires a support account with those companies as well. The applications themselves are very weird if you're not used to living in a mainframe environment. For example, you can get PKZIP for z/OS, and there is a 437 page PDF describing just the error messages alone.

Generally, learning how to get around in a mainframe environment requires working in a mainframe shop, nearly all of which are very mature and have converged on certain ways of dealing with testing, deployment, upgrades, migrations, etc. These procedures are usually not documented because they're simply well-understood by all the staff, or it would be dangerous for them to be working in the mainframe environment. Learning to use just ISPF, the closest thing z/OS has to a management shell, requires not only learning ISPF, but also all the installation-specific details for a given environment.

z/OS Library

https://www-40.ibm.com/servers/resourcelink/svc00100.nsf/pag...

There's also a z/VM library.

You can fill a large room with the printed manuals.

When I was working on the mainframe, I usually had about 20' of shelves for the manuals I used.

The trick is knowing where to look in which manual.

I characterised my job as being a professional [IBM] manual reader.

There was a gazillion times a programmer came to me with a question and I pulled out a manual.

Source code was also available for many pieces. I made several modifications and patches.

Everything was books. There weren't very many of the computers anyway.

Also maybe you should know about Seattle's 'Living Computer History Museum'. It's been closed for a couple years but I'm ready to visit once they reopen.

There is plenty of info I thought? The platform is z/os and JCL is basically what you need to learn aside from assembly, cobol,etc...

https://www.ibm.com/docs/en/zos-basic-skills?topic=sdsf-what...

https://www.ibm.com/docs/en/zos-basic-skills?topic=zos-assem...

Pub400.com – Your public IBM I 7.5 server

https://news.ycombinator.com/item?id=33032095 2022-09-30

I haven't touched a mainframe in over a decade(and i was only ever doing user administration on one) but from my experience with hpux and other legacy platforms a big part of the reason is that the actual information about the core platform lives in old hardcopy textbooks and paywalled vendor documentation, and because a lot of those platforms are "feature frozen" the old textbooks are still valid enough.

And on the application side you are typically dealing with databases and Runtimes(Java mostly) that also exist on Linux and have an large Linux centric community where any legacy programmers and admins kind of drown in the masses.

There is a whole bunch of companies out there with large COBOL and ALGOL code-bases but but as the problem here is less about generic mainframe skills and more about knowing how to work large COBOL or ALGOL projects.

I worked on these systems at IBM. The reason they are not documented on the net is that IBM has more thorough and useful documentation on its own internal network and in it's "pubs" (copyrighted and available only to customers).

IBM also has very advanced diagnostics tools and error tracking and those were in production use before the internet came along. It makes sense to use those resources rather than the web.

This doesn't answer your question completely, but the user group communities are very strong for both IBM Mainframe and IBM i platforms. https://www.share.org/ and https://www.gse.org for example.

This is typical in other fields of engineering. Details and best methods for running advanced design tools, some of the most advanced engineering knowledge, etc, are not found online in a easily-digestible Stack Overflow format. This is the norm, software engineering/Stack Overflow are the exception.

See if your local library card gives you access to /learning-oreilly-com

There is lots of content there.

I was an IBM mainframe systems programmer (VM/370) for years.

All of the information you need is in the manuals. IBM is exceptionally good at documentation.

Also, the hardware instruction set is set in stone.

https://en.m.wikipedia.org/wiki/Hercules_(emulator)

https://www.redbooks.ibm.com

a lot of howto books

I used to be the community manager for a StackOverflow-style (pre-dated SO by a number of years) site that catered to enterprise technology and we had ~200k-400k monthly visitors just on AS/400 topics — it was a huge and active community, but there was very little advertiser interest in it. A few years after I left, they completely shuttered the site and nuked the archives, sadly, so that's probably part of it — my site and others like it were bought up, monetized, then nuked when the money moved elsewhere.

The other partial reason is that a ton of those conversations happen in old email groups that aren't indexed or index a lot worse than spam sites do. So if you're interested, searching around for mailing lists, users groups, and old forums and there's still good material out there, just buried and mostly catering to folks who know the basics, so a bit tough to parse if you're new to it.

my theory:

- incredibly proprietary

- heavily customized for the environments they are running in (and boy howdy are there many)

- COBOL/Fortran/4GL not popular or desirable (for younger developers)

1. Google doesn't want you to program for mainframes. Google wants you to program for the cloud.

2. developerworks site on IBM had some info on mainframes and mainframe programming.

3. It is not "cool".

Obligatory mention of ‘Master the Mainframe’

Just do a search at Amazon Books for: IBM Mainframe or z/OS or etc.

Designed their model to resemble retail distributorships and then shocked pickachu when they realize software doesn't need the same logistics.