"remove line breaks? where have you been all my life!"
September 17, 2014 10:36 PM Subscribe
Welcome to TextMechanic.com! A suite of simple, single task, browser based, text manipulation tools.
Basic Text Tools:
Add Prefix/Suffix into Line - Insert a prefix and/or suffix into the content of each line.
Add/Remove Line Breaks - Add new line breaks and/or remove exisiting line breaks within your text's formatting.
Count Characters, Words, Lines - Count your text's characters, words, sentences, lines and word frequency.
Delimited Column Extractor - Extract a specific column of delimited text from each line of your input text.
Find and Replace Text - Find and replace text matching your search criteria.
Letter Case Converter - Convert your text's letter case to UPPER, lower, RaNdOM and more.
Merge Text (Line by Line) - Merge two sets of text line by line with the option of writing a prefix, divider or suffix into each merged line.
Remove Duplicate Lines - Remove/delete all duplicate lines within your text/list.
Remove Empty Lines - Remove/delete all empty lines within your text/list.
Remove Extra Spaces - Remove leading/trailing/extra/all whitespaces from your text.
Remove Letter Accents - Remove common letter accents from your text. (e.g. will convert into a)
Remove Lines Containing... - Remove lines containing or not containing your specified search text.
Sort Text Lines - Sort your text's lines in alphabetical, length, random or reverse order.
Obfuscation Tools:
ASCII, Hex, Unicode, Base64, Binary, Octal Converter - Encode/decode ascii, hex, unicode, base64, binary, octal.
Binary Code Translator - Obfuscate text by encoding it into the 0s and 1s of binary code. Very geeky!
Disemvowel Tool - Remove vowels "aeiou" or any other set of letters from text.
Encryption Generator - Encrypt/decrypt text via password using Tiny Encryption Algorithm (TEA) and base64.
Reverse Text Generator - Reverse text, flip text, reverse wording, flip wording, reverse each word's lettering, flip letters upside down.
ROT13 Caesar Cipher - Encrypt plaintext into ROT13 ciphertext or decrypt ROT13 ciphertext into plaintext.
Word Scrambler/Unscrambler - Scramble/unscramble each word's lettering within a body of text.
Randomization Tools:
Random Line Picker - Pick random lines from your input text/list. Lines can contain names, numbers, etc.
Random Number Generator - Generate random numbers from your entered low/high range with prefix, suffix and delimiter options.
Random String Generator - Generate random text string(s) from your entered input elements such as characters, words, sentences, etc.
String Randomizer - Randomize text strings with the option of a delimiter.
Combination / Permutation Tools:
Combination Generator - Make combination from text.
Line Combination Generator - Make combination from lines of text.
Permutation Generator - Make permutations from text. (Letters, numbers, symbols, words, sentences, etc.)
Numeration Tools:
Generate List of Numbers - Generate a list of sequential numbers from your selected low/high number range.
Number Each Line - Add a sequential number to each line of text. Enumerate items within a list.
Online Tally Counter - Count using multiple, indepently named tallies which are recorded into a printable list.
Basic Text Tools:
Add Prefix/Suffix into Line - Insert a prefix and/or suffix into the content of each line.
Add/Remove Line Breaks - Add new line breaks and/or remove exisiting line breaks within your text's formatting.
Count Characters, Words, Lines - Count your text's characters, words, sentences, lines and word frequency.
Delimited Column Extractor - Extract a specific column of delimited text from each line of your input text.
Find and Replace Text - Find and replace text matching your search criteria.
Letter Case Converter - Convert your text's letter case to UPPER, lower, RaNdOM and more.
Merge Text (Line by Line) - Merge two sets of text line by line with the option of writing a prefix, divider or suffix into each merged line.
Remove Duplicate Lines - Remove/delete all duplicate lines within your text/list.
Remove Empty Lines - Remove/delete all empty lines within your text/list.
Remove Extra Spaces - Remove leading/trailing/extra/all whitespaces from your text.
Remove Letter Accents - Remove common letter accents from your text. (e.g. will convert into a)
Remove Lines Containing... - Remove lines containing or not containing your specified search text.
Sort Text Lines - Sort your text's lines in alphabetical, length, random or reverse order.
Obfuscation Tools:
ASCII, Hex, Unicode, Base64, Binary, Octal Converter - Encode/decode ascii, hex, unicode, base64, binary, octal.
Binary Code Translator - Obfuscate text by encoding it into the 0s and 1s of binary code. Very geeky!
Disemvowel Tool - Remove vowels "aeiou" or any other set of letters from text.
Encryption Generator - Encrypt/decrypt text via password using Tiny Encryption Algorithm (TEA) and base64.
Reverse Text Generator - Reverse text, flip text, reverse wording, flip wording, reverse each word's lettering, flip letters upside down.
ROT13 Caesar Cipher - Encrypt plaintext into ROT13 ciphertext or decrypt ROT13 ciphertext into plaintext.
Word Scrambler/Unscrambler - Scramble/unscramble each word's lettering within a body of text.
Randomization Tools:
Random Line Picker - Pick random lines from your input text/list. Lines can contain names, numbers, etc.
Random Number Generator - Generate random numbers from your entered low/high range with prefix, suffix and delimiter options.
Random String Generator - Generate random text string(s) from your entered input elements such as characters, words, sentences, etc.
String Randomizer - Randomize text strings with the option of a delimiter.
Combination / Permutation Tools:
Combination Generator - Make combination from text.
Line Combination Generator - Make combination from lines of text.
Permutation Generator - Make permutations from text. (Letters, numbers, symbols, words, sentences, etc.)
Numeration Tools:
Generate List of Numbers - Generate a list of sequential numbers from your selected low/high number range.
Number Each Line - Add a sequential number to each line of text. Enumerate items within a list.
Online Tally Counter - Count using multiple, indepently named tallies which are recorded into a printable list.
I am filled with the overpowering urge to snark here along the lines of "those who will not learn from Unix tools are condemned to reinvent them, less flexibly" but, hey, different strokes for different folks -- I'm sure quite a few will find this useful.
Nevertheless, tools to do virtually all of this, quite easily, have existed for decades. So if this is the sort of thing you find yourself needing to do often.. when you outgrow the browser-based approach, or don't want to do it manually for 500 different files, the magical and surprisingly elegant world of the command line will be waiting for you..
posted by Nerd of the North at 10:41 PM on September 17, 2014 [33 favorites]
Nevertheless, tools to do virtually all of this, quite easily, have existed for decades. So if this is the sort of thing you find yourself needing to do often.. when you outgrow the browser-based approach, or don't want to do it manually for 500 different files, the magical and surprisingly elegant world of the command line will be waiting for you..
posted by Nerd of the North at 10:41 PM on September 17, 2014 [33 favorites]
Not sure a recreation of the main link was necessary, but it is a nice set of tools. Thanks!
posted by me3dia at 11:08 PM on September 17, 2014
posted by me3dia at 11:08 PM on September 17, 2014
I came in to say more or less the same thing as Nerd of the North. I pity people stuck in a Windows environment. So much of this is so easy straight from the shell in most flavors of Unix-like OSes. Need to do some tasks all the time? Just write some scripts.
For those who saw my MeTa: yes, I'm using Lynx right now.
Also, for those who could use this all the time who are stuck with Windows at work: check out Cygwin.
posted by [expletive deleted] at 11:22 PM on September 17, 2014 [6 favorites]
For those who saw my MeTa: yes, I'm using Lynx right now.
Also, for those who could use this all the time who are stuck with Windows at work: check out Cygwin.
posted by [expletive deleted] at 11:22 PM on September 17, 2014 [6 favorites]
I mean the shell environment is great, but sometimes you are on someone else's Windows box without admin rights, or at a library, or in a situation where all you have access to is a browser. This looks useful for those cases.
posted by en forme de poire at 11:31 PM on September 17, 2014 [9 favorites]
posted by en forme de poire at 11:31 PM on September 17, 2014 [9 favorites]
...the magical and surprisingly elegant world of the command line will be waiting for you..
LOL -- I read that to myself in Comic Book Guy voice and almost choked.
posted by Clave at 12:01 AM on September 18, 2014 [20 favorites]
LOL -- I read that to myself in Comic Book Guy voice and almost choked.
posted by Clave at 12:01 AM on September 18, 2014 [20 favorites]
I laugh with equal derision at the cretinous fools who require gaudy ornate multi-line text editors like
posted by XMLicious at 12:04 AM on September 18, 2014 [8 favorites]
emacs
or vi
as a crutch for their lame grasp of the command line! Who needs to edit more than one line of a text file at a time, really, or any of the other features of those decadent wastes of hard drive space, when you could just write a bunch of scripts to do it? Even sed
is rather effete and dissipated - ed
is the only text editor for me!posted by XMLicious at 12:04 AM on September 18, 2014 [8 favorites]
I didn't mean to say this isn't a useful site. In a lot of ways, I think it's pretty great and a good example of classic "best of the web" FPP material. As someone who only truly discovered the power of the shell less than five years ago, I definitely cop to having some lingering zeal of the newly converted.
On preview, some mockery is probably deserved here.
posted by [expletive deleted] at 12:13 AM on September 18, 2014 [3 favorites]
On preview, some mockery is probably deserved here.
posted by [expletive deleted] at 12:13 AM on September 18, 2014 [3 favorites]
I definitely cop to having some lingering zeal of the newly converted.
June 24, 1995
posted by mikelieman at 1:38 AM on September 18, 2014 [1 favorite]
June 24, 1995
posted by mikelieman at 1:38 AM on September 18, 2014 [1 favorite]
I suppose you'll be wanting a free text editor next.
">'>HTML Editr App I suppose some credit should go to Internet Letterbock's Top Tips
MetaFilter's parser really doesn't like to preview that URL...
posted by davemee at 1:53 AM on September 18, 2014 [2 favorites]
">'>HTML Editr App I suppose some credit should go to Internet Letterbock's Top Tips
MetaFilter's parser really doesn't like to preview that URL...
posted by davemee at 1:53 AM on September 18, 2014 [2 favorites]
I broke MetaFilter's parser and I look a fool. Do I get animated gifs now?
posted by davemee at 1:55 AM on September 18, 2014
posted by davemee at 1:55 AM on September 18, 2014
While all of this and more could be accomplished from the command line, having a bookmark like this handy to just fix the case of badly capitalized article titles, etc. is likely to be much more useful.
posted by ob1quixote at 2:26 AM on September 18, 2014
posted by ob1quixote at 2:26 AM on September 18, 2014
> I pity people stuck in a Windows environment. So much of this is so easy straight from the shell in most flavors of Unix-like OSes.
All the massed billions of computer users, crying out daily for some easy method to pick random lines out of a text file from a command prompt! If only they would see the light!
Your own use case might mean that an OS where you can knock up scripts to do this stuff at a command prompt is preferable. I would contend that this is a niche requirement. In any event, it's equally do-able on Windows with Powershell.
posted by GallonOfAlan at 3:33 AM on September 18, 2014 [1 favorite]
All the massed billions of computer users, crying out daily for some easy method to pick random lines out of a text file from a command prompt! If only they would see the light!
Your own use case might mean that an OS where you can knock up scripts to do this stuff at a command prompt is preferable. I would contend that this is a niche requirement. In any event, it's equally do-able on Windows with Powershell.
posted by GallonOfAlan at 3:33 AM on September 18, 2014 [1 favorite]
This tool is built-with and functions-in Client Side JavaScripting, so only your computer will see or process your data input/output.
posted by Obscure Reference at 4:36 AM on September 18, 2014 [2 favorites]
posted by Obscure Reference at 4:36 AM on September 18, 2014 [2 favorites]
> it's equally do-able on Windows with Powershell.
I'm sure it is, but has Powershell been around for the 25+ years that the Unix has been for me? Will it be around for another quarter century, essentially unchanged so you never have to learn another way of doing things again?
posted by scruss at 5:18 AM on September 18, 2014 [1 favorite]
I'm sure it is, but has Powershell been around for the 25+ years that the Unix has been for me? Will it be around for another quarter century, essentially unchanged so you never have to learn another way of doing things again?
posted by scruss at 5:18 AM on September 18, 2014 [1 favorite]
Smug bash ninjas, could you direct the poor plebs to a straightforward place to learn how to do this stuff as quickly and easily as you make it sound, or at least provide a couple of code examples?
posted by Magnakai at 5:47 AM on September 18, 2014 [7 favorites]
posted by Magnakai at 5:47 AM on September 18, 2014 [7 favorites]
This is excellent!
posted by royalsong at 6:19 AM on September 18, 2014 [1 favorite]
posted by royalsong at 6:19 AM on September 18, 2014 [1 favorite]
A long time ago one could use a sweet Mac OS 8 program called Torquemada the Inquisitor by Greg Swann to manipulate text in ways that just weren't available otherwise. I discovered it while working at a pre-press shop, and later came to think of it as Unix shell tools (grep, mostly) implemented inside a little Mac program.
Hah! Here is a history of the program, from 1992…and here is a blog post explaining how OS X 10.5 finally dropped support for Classic apps and thereby finally killed Torquemada. *sniffle*
Anyway, my point is that sometimes you really don't have the best tool for the job immediately to hand -- and we shouldn't sneer at someone who has decided never to have that feeling again. :7)
posted by wenestvedt at 7:24 AM on September 18, 2014
Hah! Here is a history of the program, from 1992…and here is a blog post explaining how OS X 10.5 finally dropped support for Classic apps and thereby finally killed Torquemada. *sniffle*
Anyway, my point is that sometimes you really don't have the best tool for the job immediately to hand -- and we shouldn't sneer at someone who has decided never to have that feeling again. :7)
posted by wenestvedt at 7:24 AM on September 18, 2014
Magnakai, you can try these "useful unix commands for exploring data" or the more gentle "Unix for Poets" (PDF).
posted by wenestvedt at 7:27 AM on September 18, 2014 [6 favorites]
posted by wenestvedt at 7:27 AM on September 18, 2014 [6 favorites]
So much of this is so easy straight from the shell in most flavors of Unix-like OSes.
I'm a unix wonk as much as any here but don't trivialize the complexities of text processing. Small challenge, fine a non-trivial LibraOffice document, say a few hundred pages, pipe it into standard unix text commands to remove duplicate sentences and oh say the vowels and print it directly with all the original formatting. Yep, good luck with that.
Plain text is great, and core grasp of passing formatted text through a text editor to clean it up (paste into notepad) would help a lot of folks.
This would be a vastly more useful little web tool if the menu didn't reset the text so one could 'chain' a sequence of functions on the same file.
posted by sammyo at 7:29 AM on September 18, 2014 [3 favorites]
I'm a unix wonk as much as any here but don't trivialize the complexities of text processing. Small challenge, fine a non-trivial LibraOffice document, say a few hundred pages, pipe it into standard unix text commands to remove duplicate sentences and oh say the vowels and print it directly with all the original formatting. Yep, good luck with that.
Plain text is great, and core grasp of passing formatted text through a text editor to clean it up (paste into notepad) would help a lot of folks.
This would be a vastly more useful little web tool if the menu didn't reset the text so one could 'chain' a sequence of functions on the same file.
posted by sammyo at 7:29 AM on September 18, 2014 [3 favorites]
There's a whole world of "Unix tools wrapped up in a browser app" out there. It's not a bad approach. I love my Unix command line too, but sometimes it's just simpler to paste some shit into a web page with a decent UI. And for 99% of the world it's entirely better.
posted by Nelson at 7:38 AM on September 18, 2014
posted by Nelson at 7:38 AM on September 18, 2014
Smug bash ninjas, could you direct the poor plebs to a straightforward place to learn how to do this stuff as quickly and easily as you make it sound, or at least provide a couple of code examples?
Wenestvedt's links are a good starting point. From there, the man pages can be useful once you learn to interpret their koans.
One big thing Unix tools have over this site is support for regular expressions. My past few jobs featured lots of parsing and editing of text files, and learning regular expressions more or less saved my sanity.
As requested, an example: taking the text of this thread, numbering lines and trimming whitespace:
And one other thing I wish this this tool did was remove newlines, but leave one newline intact where they are doubled up. For that:
Anyway, I'm not denying the utility of Text Mechanic, and I'm not trying to deride people for whom it would be useful. Hell, not long ago, I was more or less in the same boat. All I'm saying is anyone who finds themselves using something like this all the time for the same task would probably be well served by the shell environment and its assortment of tools.
posted by [expletive deleted] at 7:57 AM on September 18, 2014 [2 favorites]
Wenestvedt's links are a good starting point. From there, the man pages can be useful once you learn to interpret their koans.
One big thing Unix tools have over this site is support for regular expressions. My past few jobs featured lots of parsing and editing of text files, and learning regular expressions more or less saved my sanity.
As requested, an example: taking the text of this thread, numbering lines and trimming whitespace:
lynx -dump http://www.metafilter.com/142853/ | nl | sed -r 's/(^\s+|\s$)//' >thread.txt
And one other thing I wish this this tool did was remove newlines, but leave one newline intact where they are doubled up. For that:
perl -pi -e 's/(.)\n/\1/' thread.txt
Anyway, I'm not denying the utility of Text Mechanic, and I'm not trying to deride people for whom it would be useful. Hell, not long ago, I was more or less in the same boat. All I'm saying is anyone who finds themselves using something like this all the time for the same task would probably be well served by the shell environment and its assortment of tools.
posted by [expletive deleted] at 7:57 AM on September 18, 2014 [2 favorites]
LOL -- I read that to myself in Comic Book Guy voice and almost choked.
Worst. Throat obstruction. Ever.
posted by Celsius1414 at 8:06 AM on September 18, 2014 [1 favorite]
Worst. Throat obstruction. Ever.
posted by Celsius1414 at 8:06 AM on September 18, 2014 [1 favorite]
> This would be a vastly more useful little web tool if the menu didn't reset the text so one could 'chain' a sequence of functions on the same file.
I guess I have a new project to work on. :)
posted by Condroidulations! at 8:32 AM on September 18, 2014
I guess I have a new project to work on. :)
posted by Condroidulations! at 8:32 AM on September 18, 2014
My first experience with regular expressions was with an OS called Domain, which was no Unix, but something designed with the knowledge of Unix, so it had a lot of improvements. I learned it first, then later used Unix.
This may be nostalgia talking, since I don't remember how their regular expressions worked, but I remember learning Unix's and thinking that its regular expressions are so much more convoluted, unintuitive, and inflexible compared to Domain's that I decided I hated Unix. I say this as I type on my Linux machine, because HP basically dropped the whole OS after they bought the company and went with Unix, so I don't have a choice.
I say this to make the point that if you are used to something, like Unix's regular expressions, and there's nothing better out there (which there isn't today), then you might get the mistaken impression that it is the greatest thing in the world. It isn't. Not by a long shot. But it's everywhere and it's the best we've got so we're kind of stuck with it.
posted by eye of newt at 8:44 AM on September 18, 2014 [2 favorites]
This may be nostalgia talking, since I don't remember how their regular expressions worked, but I remember learning Unix's and thinking that its regular expressions are so much more convoluted, unintuitive, and inflexible compared to Domain's that I decided I hated Unix. I say this as I type on my Linux machine, because HP basically dropped the whole OS after they bought the company and went with Unix, so I don't have a choice.
I say this to make the point that if you are used to something, like Unix's regular expressions, and there's nothing better out there (which there isn't today), then you might get the mistaken impression that it is the greatest thing in the world. It isn't. Not by a long shot. But it's everywhere and it's the best we've got so we're kind of stuck with it.
posted by eye of newt at 8:44 AM on September 18, 2014 [2 favorites]
eye of newt: you are literally the first person I've ever heard say nice things about Domain/OS. I used to curse it because it was more-or-less Unix but just different enough not to fit in with our other BSD machines. The decision to make // in pathnames be special really confounded me.
But I'm fascinated by your regexp story. Do you have a reference on how those regexps worked? I found some docs on AEGIS regular expression mode, is that the syntax you fondly recall? It looks pretty much similar to the regexp used in BSD sed and the like, just cosmetic differences like % instead of ^ and ? instead of .
posted by Nelson at 9:07 AM on September 18, 2014
But I'm fascinated by your regexp story. Do you have a reference on how those regexps worked? I found some docs on AEGIS regular expression mode, is that the syntax you fondly recall? It looks pretty much similar to the regexp used in BSD sed and the like, just cosmetic differences like % instead of ^ and ? instead of .
posted by Nelson at 9:07 AM on September 18, 2014
> I mean the shell environment is great, but sometimes you are on someone else's Windows box without admin rights, or at a library, or in a situation where all you have access to is a browser.
It's funny how some things actually have moved to the cloud. Used to be when I wanted a special character I'd have to find it in the Character Map program (Windows). Now, I just google e.g. "unicode epsilon" and cut and paste it from the first page returned. (Java's string.toLowerCase() is usually where it shows up.) It just so much faster.
posted by benito.strauss at 9:27 AM on September 18, 2014 [3 favorites]
It's funny how some things actually have moved to the cloud. Used to be when I wanted a special character I'd have to find it in the Character Map program (Windows). Now, I just google e.g. "unicode epsilon" and cut and paste it from the first page returned. (Java's string.toLowerCase() is usually where it shows up.) It just so much faster.
posted by benito.strauss at 9:27 AM on September 18, 2014 [3 favorites]
OK nerds. I love you guys, really, but there is literally no need to come in here and pee in the pool. Some of us are not interested in shell environments, or using linux, command lines, etc. Our skillsets lie in content, not code.
So this is awesome, thanks for posting!
/gets out more chlorine.
posted by Kimberly at 10:09 AM on September 18, 2014 [4 favorites]
So this is awesome, thanks for posting!
/gets out more chlorine.
posted by Kimberly at 10:09 AM on September 18, 2014 [4 favorites]
I think the pro-unix crowd (of which I am one) are simply responding to the title of the post. Because removing line breaks from text has probably been around longer than the poster. :) That being said, these are pretty awesome for people who don't have access to a terminal.
posted by thewalledcity at 10:18 AM on September 18, 2014 [1 favorite]
posted by thewalledcity at 10:18 AM on September 18, 2014 [1 favorite]
UNIX tools are very powerful, but they can also be a little... less than straightforward. I can do every one of these things at the command line, and I prefer to do so because I'm used to it and don't have an issue with the mental model of the shell and everything above it and underneath it (i.e. I know it a bit *too* well to recommend it as the best tool for everyone :-))
However, using the command line for text processing kind of sucks for most people. Just to mention one potential deal breaker (there are others): How all the various symbols interact and how shell escaping works. It's all wonderful until you want to use a space in your filename, or search for '?' or '[tag]' using a regex. And this is (somewhat) simple stuff for those who know it, but why should someone have to know it?
There's also the issue of discoverability. I would consider myself an expert, but even I don't know every single option to every single command or even all of the commands out there (I didn't know about "nl" above, I generally use "cat -n"). "apropos" is... spectacularly ineffective unless you know what you're doing.
The ideal tool in my mind is basically Automator for text processing. I feel like this should already exist somewhere -- and I suppose you could write the automator plugins to make it happen (although I worry if Apple is going to kill it off someday). That would bring the "full power of the command line" to people who just want to deal with parsing their text, not some random regex or scripting language.
posted by smidgen at 11:32 AM on September 18, 2014 [1 favorite]
However, using the command line for text processing kind of sucks for most people. Just to mention one potential deal breaker (there are others): How all the various symbols interact and how shell escaping works. It's all wonderful until you want to use a space in your filename, or search for '?' or '[tag]' using a regex. And this is (somewhat) simple stuff for those who know it, but why should someone have to know it?
There's also the issue of discoverability. I would consider myself an expert, but even I don't know every single option to every single command or even all of the commands out there (I didn't know about "nl" above, I generally use "cat -n"). "apropos" is... spectacularly ineffective unless you know what you're doing.
The ideal tool in my mind is basically Automator for text processing. I feel like this should already exist somewhere -- and I suppose you could write the automator plugins to make it happen (although I worry if Apple is going to kill it off someday). That would bring the "full power of the command line" to people who just want to deal with parsing their text, not some random regex or scripting language.
posted by smidgen at 11:32 AM on September 18, 2014 [1 favorite]
"remove line breaks? where have you been all my life!"
Um, OS X - and of course other *nixes. The Mac OS has had system-wide text manipulation plug-ins for quite a while (called Services), so that commands could be executed by contextual menu, but it seems they built it into the system recently. Certain unix-like text commands have always worked, like: ctrl-a = brings cursor to the beginning of a line, and so on. I know about eight of them, there may be more. These commands work in any app that handles text.
But yeah, I understand this is coming to Windows for the first time, the OS most people use, and that's a good thing.
posted by sudon't at 11:40 AM on September 18, 2014
Um, OS X - and of course other *nixes. The Mac OS has had system-wide text manipulation plug-ins for quite a while (called Services), so that commands could be executed by contextual menu, but it seems they built it into the system recently. Certain unix-like text commands have always worked, like: ctrl-a = brings cursor to the beginning of a line, and so on. I know about eight of them, there may be more. These commands work in any app that handles text.
But yeah, I understand this is coming to Windows for the first time, the OS most people use, and that's a good thing.
posted by sudon't at 11:40 AM on September 18, 2014
I think the pro-unix crowd (of which I am one) are simply responding to the title of the post. Because removing line breaks from text has probably been around longer than the poster.
The title quote is actually from me.
Hi, I have to interact with a bunch of people who have made choices in their life that means they are using Windows or OSX machines and we have to swap documents. I can also do things on the command line. There are ways to feel good about myself by sekritly knowing that everyone else is doing it wrong, sure, but I also make part of my living writing for pay and those documents need to come in a few standard formats that you've probably heard of. And while I might bitch about that a bit offline, I'd rather get paid to write than be a nerd stereotype.
These tools solved problems for me because I was otherwise running annoying find/replace scripts on other people's Word documents so that I could format switch them (say, to put inside a CMS so I could put them on the website) in order to do the things I wanted to do with them. Part of the reasons that a lot of people love technology is that it solves problems for them without them having to deal with awkward IT people who make them feel bad about the choices that they've made and the way they've decided to structure their technological lives.
The right technological solution is one that works for you and if you don't want to see me using Comic Sans in my library flyers, don't look. Understand that it's a taste issue. The biggest digital divide issue we're facing now is people's technological trepidation about feeling stupid, not understanding how to do things, and being treated badly by other people when they try to learn, especially when they have trouble. This group of people, people stuck in what we call the empowerment divide, is nearly as large as the people who can't afford the technology or who can't get access from where they live. Finding solutions for them is part of solving the larger problem.
Don't contribute to that. Be helpful. Most people in the world never see a command line and I don't think there's a compelling argument--if you're really meeting people on their terms not yours--that they should. These are nice tools.
posted by jessamyn at 1:36 PM on September 18, 2014 [27 favorites]
The title quote is actually from me.
Hi, I have to interact with a bunch of people who have made choices in their life that means they are using Windows or OSX machines and we have to swap documents. I can also do things on the command line. There are ways to feel good about myself by sekritly knowing that everyone else is doing it wrong, sure, but I also make part of my living writing for pay and those documents need to come in a few standard formats that you've probably heard of. And while I might bitch about that a bit offline, I'd rather get paid to write than be a nerd stereotype.
These tools solved problems for me because I was otherwise running annoying find/replace scripts on other people's Word documents so that I could format switch them (say, to put inside a CMS so I could put them on the website) in order to do the things I wanted to do with them. Part of the reasons that a lot of people love technology is that it solves problems for them without them having to deal with awkward IT people who make them feel bad about the choices that they've made and the way they've decided to structure their technological lives.
The right technological solution is one that works for you and if you don't want to see me using Comic Sans in my library flyers, don't look. Understand that it's a taste issue. The biggest digital divide issue we're facing now is people's technological trepidation about feeling stupid, not understanding how to do things, and being treated badly by other people when they try to learn, especially when they have trouble. This group of people, people stuck in what we call the empowerment divide, is nearly as large as the people who can't afford the technology or who can't get access from where they live. Finding solutions for them is part of solving the larger problem.
Don't contribute to that. Be helpful. Most people in the world never see a command line and I don't think there's a compelling argument--if you're really meeting people on their terms not yours--that they should. These are nice tools.
posted by jessamyn at 1:36 PM on September 18, 2014 [27 favorites]
I, the OP, have been using computers in one way or another since the late seventies, and I thank you nerds every day for learning unix and all that... SO THAT I NEVER HAD TO.
I'm even lazier than all y'all programmer people! That's why I posted this.
To the unlearnt masses: I hope you appreciate this site as much as I do! To the nerds: thanks, I hope you end up writing something that improves upon this. I'll be among the first in line to reap the benefits of your hard work.
posted by not_on_display at 2:04 PM on September 18, 2014 [3 favorites]
I'm even lazier than all y'all programmer people! That's why I posted this.
To the unlearnt masses: I hope you appreciate this site as much as I do! To the nerds: thanks, I hope you end up writing something that improves upon this. I'll be among the first in line to reap the benefits of your hard work.
posted by not_on_display at 2:04 PM on September 18, 2014 [3 favorites]
Sigh.
Can we stop enumerating all of the nerdy ways to say 'meh'? Unless you're prepared to personally go install Cygwin and a suite of canned *.awk files on all the machines where people might access this web page, all y'alls impressive bash-fu really isn't a substitute for the site in the post.
Because, while all our algorithmic and computational power is impressive, it's really only useful to people as far as we deliver it to them in a way they can use. Everyone has a javascript engine on every device these days, so making these available in a browser is a big win.
At a previous company I once spent a lot of time indexing and writing search tools for a 5-year archive of messages. It was cool and all, but you had to remote in to one particular machine and access all the info from within a terminal. At someone's suggestion I put up an Apache server and turned the search script into the simplest of all CGI scripts, and suddenly everyone in the office was 5 seconds away from all kinds of great information. They loved it and I love seeing the tools I create get used.
As a gracious nerd you learn to accept all the compliments for the last few steps, and not complain about the 26 miles you had to run to get there.
posted by benito.strauss at 4:44 PM on September 18, 2014 [4 favorites]
Can we stop enumerating all of the nerdy ways to say 'meh'? Unless you're prepared to personally go install Cygwin and a suite of canned *.awk files on all the machines where people might access this web page, all y'alls impressive bash-fu really isn't a substitute for the site in the post.
Because, while all our algorithmic and computational power is impressive, it's really only useful to people as far as we deliver it to them in a way they can use. Everyone has a javascript engine on every device these days, so making these available in a browser is a big win.
At a previous company I once spent a lot of time indexing and writing search tools for a 5-year archive of messages. It was cool and all, but you had to remote in to one particular machine and access all the info from within a terminal. At someone's suggestion I put up an Apache server and turned the search script into the simplest of all CGI scripts, and suddenly everyone in the office was 5 seconds away from all kinds of great information. They loved it and I love seeing the tools I create get used.
As a gracious nerd you learn to accept all the compliments for the last few steps, and not complain about the 26 miles you had to run to get there.
posted by benito.strauss at 4:44 PM on September 18, 2014 [4 favorites]
Nelson wrote:
posted by Nerd of the North at 5:00 PM on September 18, 2014 [1 favorite]
you are literally the first person I've ever heard say nice things about Domain/OS. I used to curse it because it was more-or-less Unix but just different enough not to fit in with our other BSD machines. The decision to make // in pathnames be special really confounded me.Noooo! Domain/OS was AWESOME.. for its time. It wasn't without its issues but for the mid-to-late-80s it had some amazing features that would take similar operating systems years to incorporate. The // thing was the key to a fully transparent networked filesystem that was amazingly usable at a time when NFS was a painful joke (there's room to disagree whether NFS was ever anything other, but the point is the Apollo filesystem worked.) The "edit pads" integrated into the windowing system were pretty cool, too, with surprisingly advanced editing features and terrific integration with the extended keyboard, which supported many common editor functions directly via custom keys. And finally, I still wish that other Unix-likes had picked up the idea of allowing environment variables in symbolic links. That was genuinely useful.
posted by Nerd of the North at 5:00 PM on September 18, 2014 [1 favorite]
I'll bet Plan 9 has something equivalent to environment variables in symbolic links. They probably can also be URLs. In Unicode.
posted by benito.strauss at 5:45 PM on September 18, 2014 [1 favorite]
posted by benito.strauss at 5:45 PM on September 18, 2014 [1 favorite]
Note that DomainOS was active about when most everyone else was thrilled to have windows 3.1. It was amazing but a bit too proprietary without a Steve Jobs to build excitement. First time I saw how to attach a symbolic debugger to a live running program at a remote location over the net. Latency was almost non-existent, click on a file on a system in Europe and it opened before your finger lifted. They bet on a technology called token-net that has not been heard of for years. But from murmurs it was financial shenanigans that actually killed the company.
posted by sammyo at 5:50 PM on September 18, 2014
posted by sammyo at 5:50 PM on September 18, 2014
But I'm fascinated by your regexp story. Do you have a reference on how those regexps worked? I found some docs on AEGIS regular expression mode, is that the syntax you fondly recall? It looks pretty much similar to the regexp used in BSD sed and the like, just cosmetic differences like % instead of ^ and ? instead of--Nelson
Yeah, well if you provide an actual link to the Apollo regular expression, it doesn't look so impressive. I seem to remember it differently but that's how nostalgia works. Maybe @ just works better as an exception notifier than the backslash, or maybe there were a lot fewer exception characters so you didn't end up with unreadable expressions like this:
^[a-zA-Z0-9\',!;\?\$\^:\\\/`\|~&\" @#%\*\{}\(\)_\+\.\s=-]{1,1000}$
I found this interesting regular expression alternative while searching around. It is certainly something to consider if you are writing code for someone else or code that will be around for so long you wouldn't otherwise understand it next time you took a look.
posted by eye of newt at 8:12 PM on September 18, 2014 [5 favorites]
Yeah, well if you provide an actual link to the Apollo regular expression, it doesn't look so impressive. I seem to remember it differently but that's how nostalgia works. Maybe @ just works better as an exception notifier than the backslash, or maybe there were a lot fewer exception characters so you didn't end up with unreadable expressions like this:
^[a-zA-Z0-9\',!;\?\$\^:\\\/`\|~&\" @#%\*\{}\(\)_\+\.\s=-]{1,1000}$
I found this interesting regular expression alternative while searching around. It is certainly something to consider if you are writing code for someone else or code that will be around for so long you wouldn't otherwise understand it next time you took a look.
posted by eye of newt at 8:12 PM on September 18, 2014 [5 favorites]
Thanks for the tips.
For someone who spends a not insignificant portion of his day on the terminal, it's daunting how much there is to learn about potentially useful but arcane procedures.
Verbal Expressions sound amazing. Thanks so much for linking to them, I think that'll save a not insignificant amount of time
posted by Magnakai at 3:22 AM on September 19, 2014
For someone who spends a not insignificant portion of his day on the terminal, it's daunting how much there is to learn about potentially useful but arcane procedures.
Verbal Expressions sound amazing. Thanks so much for linking to them, I think that'll save a not insignificant amount of time
posted by Magnakai at 3:22 AM on September 19, 2014
So much of this is so easy straight from the shell in most flavors of Unix-like OSes. Need to do some tasks all the time? Just write some scripts.
Do you know why the GUI is everywhere?
Because discoverability on the command line sucks. You have a prompt aaaaaand... that's it. You want it to do anything? Buy a book. Spend some quality time with Google and innumerable tutorial pages. Learn rudimentary programming. And then commit it all to memory or copy it down to a note someplace you won't forget where it is.
Or you can go to this web tool, and see with your own eyes the operation you want, listed in plain language. Boom. Done.
Immediate productivity with minimal investment in time and mental resources. That's why the GUI is everywhere.
posted by Slap*Happy at 7:21 AM on September 19, 2014 [5 favorites]
Do you know why the GUI is everywhere?
Because discoverability on the command line sucks. You have a prompt aaaaaand... that's it. You want it to do anything? Buy a book. Spend some quality time with Google and innumerable tutorial pages. Learn rudimentary programming. And then commit it all to memory or copy it down to a note someplace you won't forget where it is.
Or you can go to this web tool, and see with your own eyes the operation you want, listed in plain language. Boom. Done.
Immediate productivity with minimal investment in time and mental resources. That's why the GUI is everywhere.
posted by Slap*Happy at 7:21 AM on September 19, 2014 [5 favorites]
Unless you're prepared to personally go install CygwinOh my god, why would you subject yourself to that?
Macs already have all of the cool stuff built-in, while GoW provides native Win32 versions of the standard *nix toolset.
posted by schmod at 12:58 PM on September 19, 2014 [2 favorites]
I laugh with equal derision at the cretinous fools who require gaudy ornate multi-line text editors like emacs or vi as a crutch for their lame grasp of the command line!
I laugh with greater derision at slack-jawed troglodytes who require command lines. I use very fine tweezers to directly manipulate RAM and for output I hooked an RS-232 port up to my tongue.
posted by ROU_Xenophobe at 1:08 PM on September 19, 2014 [2 favorites]
I laugh with greater derision at slack-jawed troglodytes who require command lines. I use very fine tweezers to directly manipulate RAM and for output I hooked an RS-232 port up to my tongue.
posted by ROU_Xenophobe at 1:08 PM on September 19, 2014 [2 favorites]
Hell, I'm whistling at 14400 baud right now.
posted by not_on_display at 7:58 PM on September 19, 2014 [1 favorite]
posted by not_on_display at 7:58 PM on September 19, 2014 [1 favorite]
SSHHFHFHFFFFCHHKHHHH
posted by en forme de poire at 8:34 PM on September 19, 2014
posted by en forme de poire at 8:34 PM on September 19, 2014
+++ATH0
posted by not_on_display at 8:45 PM on September 19, 2014
posted by not_on_display at 8:45 PM on September 19, 2014
Yeah, well if you provide an actual link to the Apollo regular expression, it doesn't look so impressive. I seem to remember it differently but that's how nostalgia works. Maybe @ just works better as an exception notifier than the backslash, or maybe there were a lot fewer exception characters so you didn't end up with unreadable expressions like this:
^[a-zA-Z0-9\',!;\?\$\^:\\\/`\|~&\" @#%\*\{}\(\)_\+\.\s=-]{1,1000}$
Okay, now you're being ridiculous. The only characters that aren't treated literally inside square brackets are the caret, hyphen, backslash and square brackets themselves. I will admit though, that lots of regexes are unreadable ciphers, especially, heaven forbid, procedurally generated ones.
posted by [expletive deleted] at 8:50 PM on September 19, 2014 [1 favorite]
^[a-zA-Z0-9\',!;\?\$\^:\\\/`\|~&\" @#%\*\{}\(\)_\+\.\s=-]{1,1000}$
Okay, now you're being ridiculous. The only characters that aren't treated literally inside square brackets are the caret, hyphen, backslash and square brackets themselves. I will admit though, that lots of regexes are unreadable ciphers, especially, heaven forbid, procedurally generated ones.
posted by [expletive deleted] at 8:50 PM on September 19, 2014 [1 favorite]
As Jessamyn pointed out, these are useful intermediates for some (many?) users. The reason that there isn't one all-encompassing GUI solution might be that to write it would need some fairly deep text processing experience, and usually people write free tools to scratch their own itches. Those with the skills already have the tools. If I had an attention span, free time and something above rudimentary programming skills, I'd get right on with doing something like this in the browser with JavaScript. Though of course finding a toolkit to build it on that would stay current and maintained for a decade to keep it useful would be hard. Unicode, markup, file types, oh my!
GUIs have no memory; they turn every job into drudge work. Have to run a process twice? You have to go through the process twice. The command line's history helps you build up a data workflow which, once it works, allows you to save it to a script and re-run it at any time. The tyranny of the GUI was recently made clear for me when a small technical shop I know asked me if I knew a quicker way of stitching together geo-referenced images. They'd been billing a week's time to log and import each tile manually to build up a map background. In about an hour, I'd shown them how to install OSGeo4W and run gdal_merge, which did the job in 15 minutes. Now, whether the time savings have been passed on to the client I don't know, but their tech is much happier ...
posted by scruss at 7:10 AM on September 20, 2014
GUIs have no memory; they turn every job into drudge work. Have to run a process twice? You have to go through the process twice. The command line's history helps you build up a data workflow which, once it works, allows you to save it to a script and re-run it at any time. The tyranny of the GUI was recently made clear for me when a small technical shop I know asked me if I knew a quicker way of stitching together geo-referenced images. They'd been billing a week's time to log and import each tile manually to build up a map background. In about an hour, I'd shown them how to install OSGeo4W and run gdal_merge, which did the job in 15 minutes. Now, whether the time savings have been passed on to the client I don't know, but their tech is much happier ...
posted by scruss at 7:10 AM on September 20, 2014
GUIs have no memory; they turn every job into drudge work. ... The command line's history helps you build up a data workflow which, once it works, allows you to save it to a script and re-run it at any time.
Microsoft Word has "Record Macro". Turn it on, mouse click just like you always have, and then view the Visual Basic program that was created. You can run that over and over. (Personally, I always turned them into Perl programs that used COM automation, because Msoft made it hard for the VB to live outside of the document.)
Most of the Office suite allowed this, though I don't think it was widely picked up by other major products.
posted by benito.strauss at 10:14 AM on September 20, 2014
Microsoft Word has "Record Macro". Turn it on, mouse click just like you always have, and then view the Visual Basic program that was created. You can run that over and over. (Personally, I always turned them into Perl programs that used COM automation, because Msoft made it hard for the VB to live outside of the document.)
Most of the Office suite allowed this, though I don't think it was widely picked up by other major products.
posted by benito.strauss at 10:14 AM on September 20, 2014
Thanks for the regex answer, eye of newt, and my apologies for tarnishing your fond memories. I was really fishing for a radically alternative regexp syntax, and you provided very nicely with those verbal expressions. Javascript people are wonderful at creating long method chaining syntax, I'm reminded a bit of chai's unit test syntax.
The key advantage of the AEGIS syntax I see is using @ for the literal escape instead of \. The problem is \ is also a string escape in many languages, so you end up having to double up the \ and get leaning toothpick syndrome, like in emacs lisp. (ie: '\\(?:[^\\']\\|\\(?:\\\\.\\)\\)*' is a real regexp real code uses in elisp.) That's why Perl and Python have special syntax for literal regexps, a kludge that might not be necessary if we used @ instead of \.
(As for why Domain/OS's choice of // being special in pathnames was bad, it's because it broke a whole lot of code where you are concatenating bits of pathnames together. Ie it's common to have shell scripts that end up creating paths like /tmp//foo.txt, and suddenly that means something different in Unix and Domain/OS. I spent many hours dealing with that crap.)
Back on the topic of this fine post, I had occasion to use the downcase convertor tool yesterday because I can never recall how to do that right in the Unix command line despite 25 years experience. So thanks! I'd have designed the Web GUI differently, with a input-only box and a separate output box that's updated live. No big deal though, it got the job done.
posted by Nelson at 10:24 AM on September 20, 2014 [2 favorites]
The key advantage of the AEGIS syntax I see is using @ for the literal escape instead of \. The problem is \ is also a string escape in many languages, so you end up having to double up the \ and get leaning toothpick syndrome, like in emacs lisp. (ie: '\\(?:[^\\']\\|\\(?:\\\\.\\)\\)*' is a real regexp real code uses in elisp.) That's why Perl and Python have special syntax for literal regexps, a kludge that might not be necessary if we used @ instead of \.
(As for why Domain/OS's choice of // being special in pathnames was bad, it's because it broke a whole lot of code where you are concatenating bits of pathnames together. Ie it's common to have shell scripts that end up creating paths like /tmp//foo.txt, and suddenly that means something different in Unix and Domain/OS. I spent many hours dealing with that crap.)
Back on the topic of this fine post, I had occasion to use the downcase convertor tool yesterday because I can never recall how to do that right in the Unix command line despite 25 years experience. So thanks! I'd have designed the Web GUI differently, with a input-only box and a separate output box that's updated live. No big deal though, it got the job done.
posted by Nelson at 10:24 AM on September 20, 2014 [2 favorites]
Nelson, it all depends on whether ASCII or Unicode is needed. I was disappointed that Gnu tr didn't do what is logical here:
echo ÂÅRDVÄRK | tr '[[:upper:]]' '[[:lower:]]'
posted by scruss at 8:03 AM on September 21, 2014
echo ÂÅRDVÄRK | tr '[[:upper:]]' '[[:lower:]]'
posted by scruss at 8:03 AM on September 21, 2014
My Ubuntu box returns ÂÅrdvÄrk. MacOS gives me âårdvärk. Yay Unix! Despite my 25 years experience I couldn't tell you what locale either system is configured to without spending half an hour researching the botch that is Unix locale management and remembering what bits of my environment that I've tweaked are relevant to this case. The Mac has SysVile roots, so everything works even more confusingly there. (512 byte blocks, you will never die.)
Anyway I can certainly RTFM to figure out tr case conversion, but what's easier: just go to the web tool, or reading a man page to rediscover tr syntax I never use? And of course I need to remember how to do this in two tools; tr can't do the trick of leaving the first word capitalized. Pretty soon I'm writing awk or perl -pie programs just to keep my snobbery of being a Unix command line expert.
posted by Nelson at 9:05 AM on September 21, 2014 [2 favorites]
Anyway I can certainly RTFM to figure out tr case conversion, but what's easier: just go to the web tool, or reading a man page to rediscover tr syntax I never use? And of course I need to remember how to do this in two tools; tr can't do the trick of leaving the first word capitalized. Pretty soon I'm writing awk or perl -pie programs just to keep my snobbery of being a Unix command line expert.
posted by Nelson at 9:05 AM on September 21, 2014 [2 favorites]
The Mac has SysVile roots, so everything works even more confusingly there.
Indeed not! It's much stranger - it's a MACH microkernel with a BSD-ish OS and userland on top. At one time they were contributing code back to the Free, Net and OpenBSD projects, not certain if they still are. It does have some SysV-ish elements, some Linuxy elements, and some crazy-pants stuff no-one else uses. Mac and Solaris/Open Indiana are some very weird Unix beasties these days.
posted by Slap*Happy at 2:34 PM on September 21, 2014
Indeed not! It's much stranger - it's a MACH microkernel with a BSD-ish OS and userland on top. At one time they were contributing code back to the Free, Net and OpenBSD projects, not certain if they still are. It does have some SysV-ish elements, some Linuxy elements, and some crazy-pants stuff no-one else uses. Mac and Solaris/Open Indiana are some very weird Unix beasties these days.
posted by Slap*Happy at 2:34 PM on September 21, 2014
Mac OS X's weird case insensitivity is, well, weird.
posted by scruss at 3:40 PM on September 21, 2014
posted by scruss at 3:40 PM on September 21, 2014
Some of us have jobs which require us to use Word in a Windows environment.
Some of us also don't feel like learning an entirely new system of finicky programming shit, because we are not programmers.
posted by showbiz_liz at 10:44 PM on September 23, 2014
Some of us also don't feel like learning an entirely new system of finicky programming shit, because we are not programmers.
posted by showbiz_liz at 10:44 PM on September 23, 2014
Some of us have jobs which require us to use Word in a Windows environment.
You wouldn't need Word if you people could be bothered to learn a proper typesetting language...
/HaMbUrGeR
posted by benzenedream at 11:57 PM on September 24, 2014 [3 favorites]
You wouldn't need Word if you people could be bothered to learn a proper typesetting language...
/HaMbUrGeR
posted by benzenedream at 11:57 PM on September 24, 2014 [3 favorites]
I've said this before, but my friend once told me she was just going to write all of her stuff in Office with full justification and the font set to Computer Modern and see if anyone actually ever caught on. And I think she's totally right, a certain type of person sees that goddamn font and instantly starts taking you more seriously. Trololol, etc.
posted by en forme de poire at 12:46 AM on September 25, 2014 [8 favorites]
posted by en forme de poire at 12:46 AM on September 25, 2014 [8 favorites]
« Older Jessica Hernandez and the Deltas | The anime of Makoto Shinkai: beautiful, somber... Newer »
This thread has been archived and is closed to new comments
word count percent
text 28 5.61%
line 16 3.21%
your 15 3.01%
of 14 2.81%
remove 14 2.81%
lines 13 2.61%
a 11 2.20%
from 11 2.20%
and 9 1.80%
into 8 1.60%
random 8 1.60%
or 7 1.40%
generator 7 1.40%
list 7 1.40%
each 7 1.40%
posted by not_on_display at 10:39 PM on September 17, 2014 [9 favorites]