I need to match all of these opening tags:

<p>
<a href="foo">

But not these:

<br />
<hr class="foo" />

I came up with this and wanted to make sure I've got it right. I am only capturing the a-z.

<([a-z]+) *[^/]*?>

I believe it says:

  • Find a less-than, then
  • Find (and capture) a-z one or more times, then
  • Find zero or more spaces, then
  • Find any character zero or more times, greedy, except /, then
  • Find a greater-than

Do I have that right? And more importantly, what do you think?

35 Answers 11

You want the first > not preceded by a /. Look here for details on how to do that. It's referred to as negative lookbehind.

However, a naïve implementation of that will end up matching <bar/></foo> in this example document

<foo><bar/></foo>

Can you provide a little more information on the problem you're trying to solve? Are you iterating through tags programatically?

1 upvote
  flag
Yep, I sure am. Determining all the tags that are currently open, then compare that against the closed tags in a separate array. RegEx hurts my brain. – Jeff

Try:

<([^\s]+)(\s[^>]*?)?(?<!/)>

It is similar to yours, but the last > must not be after a slash, and also accepts h1.

97 upvote
  flag
<a href="foo" title="5>3"> Oops </a> – Gareth
19 upvote
  flag
That is very true, and I did think about it, but I assumed the > symbol is properly escaped to &gt;. – Kobi
63 upvote
  flag
> is valid in an attribute value. Indeed, in the ‘canonical XML’ serialisation you must not use &gt;. (Which isn't entirely relevant, except to emphasise that > in an attribute value is not at all an unusual thing.) – bobince
5 upvote
  flag
@Kobi: what does the exlamation mark (the one you placed tpward the end) mean in a regexp? – Marco Demaio
6 upvote
  flag
@bobince: are u sure? I don't understand anymore, so is this valid HTML too: <div title="this tag is a <div></div>">hello</div> – Marco Demaio
3 upvote
  flag
@Marco - Negative Lookbehind – Kobi
3 upvote
  flag
@MarcoDemaio - > does not have to be escaped in an attribute value, but < does. So this is would be valid HTML: <div title="this tag is a &lt;div>&lt;/div>">hello</div> – Daniel Haley
up vote 4425 down vote accepted

You can't parse [X]HTML with regex. Because HTML can't be parsed by regex. Regex is not a tool that can be used to correctly parse HTML. As I have answered in HTML-and-regex questions here so many times before, the use of regex will not allow you to consume HTML. Regular expressions are a tool that is insufficiently sophisticated to understand the constructs employed by HTML. HTML is not a regular language and hence cannot be parsed by regular expressions. Regex queries are not equipped to break down HTML into its meaningful parts. so many times but it is not getting to me. Even enhanced irregular regular expressions as used by Perl are not up to the task of parsing HTML. You will never make me crack. HTML is a language of sufficient complexity that it cannot be parsed by regular expressions. Even Jon Skeet cannot parse HTML using regular expressions. Every time you attempt to parse HTML with regular expressions, the unholy child weeps the blood of virgins, and Russian hackers pwn your webapp. Parsing HTML with regex summons tainted souls into the realm of the living. HTML and regex go together like love, marriage, and ritual infanticide. The <center> cannot hold it is too late. The force of regex and HTML together in the same conceptual space will destroy your mind like so much watery putty. If you parse HTML with regex you are giving in to Them and their blasphemous ways which doom us all to inhuman toil for the One whose Name cannot be expressed in the Basic Multilingual Plane, he comes. HTML-plus-regexp will liquify the n​erves of the sentient whilst you observe, your psyche withering in the onslaught of horror. Rege̿̔̉x-based HTML parsers are the cancer that is killing StackOverflow it is too late it is too late we cannot be saved the trangession of a chi͡ld ensures regex will consume all living tissue (except for HTML which it cannot, as previously prophesied) dear lord help us how can anyone survive this scourge using regex to parse HTML has doomed humanity to an eternity of dread torture and security holes using regex as a tool to process HTML establishes a breach between this world and the dread realm of c͒ͪo͛ͫrrupt entities (like SGML entities, but more corrupt) a mere glimpse of the world of reg​ex parsers for HTML will ins​tantly transport a programmer's consciousness into a world of ceaseless screaming, he comes, the pestilent slithy regex-infection wil​l devour your HT​ML parser, application and existence for all time like Visual Basic only worse he comes he comes do not fi​ght he com̡e̶s, ̕h̵i​s un̨ho͞ly radiańcé destro҉ying all enli̍̈́̂̈́ghtenment, HTML tags lea͠ki̧n͘g fr̶ǫm ̡yo​͟ur eye͢s̸ ̛l̕ik͏e liq​uid pain, the song of re̸gular exp​ression parsing will exti​nguish the voices of mor​tal man from the sp​here I can see it can you see ̲͚̖͔̙î̩́t̲͎̩̱͔́̋̀ it is beautiful t​he final snuffing of the lie​s of Man ALL IS LOŚ͖̩͇̗̪̏̈́T ALL I​S LOST the pon̷y he comes he c̶̮omes he comes the ich​or permeates all MY FACE MY FACE ᵒh god no NO NOO̼O​O NΘ stop the an​*̶͑̾̾​̅ͫ͏̙̤g͇̫͛͆̾ͫ̑͆l͖͉̗̩̳̟̍ͫͥͨe̠̅s ͎a̧͈͖r̽̾̈́͒͑e n​ot rè̑ͧ̌aͨl̘̝̙̃ͤ͂̾̆ ZA̡͊͠͝LGΌ ISͮ̂҉̯͈͕̹̘̱ TO͇̹̺ͅƝ̴ȳ̳ TH̘Ë͖́̉ ͠P̯͍̭O̚​N̐Y̡ H̸̡̪̯ͨ͊̽̅̾̎Ȩ̬̩̾͛ͪ̈́̀́͘ ̶̧̨̱̹̭̯ͧ̾ͬC̷̙̲̝͖ͭ̏ͥͮ͟Oͮ͏̮̪̝͍M̲̖͊̒ͪͩͬ̚̚͜Ȇ̴̟̟͙̞ͩ͌͝S̨̥̫͎̭ͯ̿̔̀ͅ


Have you tried using an XML parser instead?


Moderator's Note

This post is locked to prevent inappropriate edits to its content. The post looks exactly as it is supposed to look - there are no problems with its content. Please do not flag it for our attention.

177 upvote
  flag
Kobi: I think it's time for me to quit the post of Assistant Don't Parse HTML With Regex Officer. No matter how many times we say it, they won't stop coming every day... every hour even. It is a lost cause, which someone else can fight for a bit. So go on, parse HTML with regex, if you must. It's only broken code, not life and death. – bobince
27 upvote
  flag
Is it possible to use RegEx to parse this answer? – Chris Porter
2 upvote
  flag
If you can't see this post, here's a screencapture of it in all its glory: imgur.com/gOPS2.png – Andrew Keeton

I suggest using QueryPath for parsing XML and HTML in PHP. It's basically much the same syntax as jQuery, only it's on the server side.

8 upvote
  flag
@Kyle—jQuery does not parse XML, it uses the client's built–in parser (if there is one). Therefore you do not need jQuery to do it, but as little as two lines of plain old JavaScript. If there is no built–in parser, jQuery will not help. – RobG
1 upvote
  flag
@RobG Actually jQuery uses the DOM, not the built-in parser. – Qix
9 upvote
  flag
@Qix—you'd better tell the authors of the documentation then: "jQuery.parseXML uses the native parsing function of the browser…". Source: jQuery.parseXML() – RobG
3 upvote
  flag
Having come here from the meme question (meta.stackexchange.com/questions/19478/the-many-memes-of-me‌​ta/…), I love that one of the answers is 'Use jQuery' – Jorn

While it is true that asking regexes to parse arbitrary HTML is like asking a beginner to write an operating system, it's sometimes appropriate to parse a limited, known set of HTML.

If you have a small set of HTML pages that you want to scrape data from and then stuff into a database, regexes might work fine. For example, I recently wanted to get the names, parties, and districts of Australian federal Representatives, which I got off of the Parliament's web site. This was a limited, one-time job.

Regexes worked just fine for me, and were very fast to set up.

98 upvote
  flag
Also, scraping fairly regularly formatted data from large documents is going to be WAY faster with judicious use of scan & regex than any generic parser. And if you are comfortable with coding regexes, way faster to code than coding xpaths. And almost certainly less fragile to changes in what you are scraping. So bleh. – Michael Johnston
188 upvote
  flag
@MichaelJohnston "Less fragile"? Almost certainly not. Regexes care about text-formatting details than an XML parser can silently ignore. Switching between &foo; encodings and CDATA sections? Using an HTML minifier to remove all whitespace in your document that the browser doesn't render? An XML parser won't care, and neither will a well-written XPath statement. A regex-based "parser", on the other hand... – Charles Duffy
31 upvote
  flag
@CharlesDuffy for an one time job it's ok, and for spaces we use \s+ – quantum
upvote
  flag
@xiaomao that works for spaces, but newlines are also valid within XML elements. – Charles Duffy
3 upvote
  flag
@CharlesDuffy \s matches space, tab, form feed, carriage return, line feed, and possible more. Unless you are using multi-line mode it works. – quantum
2 upvote
  flag
@xiaomao nod -- multi-line mode (default in many environments) was the point. Also, note that you've focused only on whitespace -- the smallest and easiest objection -- and ignored the rest. – Charles Duffy
1 upvote
  flag
@CharlesDuffy If it's the default you can always disable it... If I was to use this kind of regex I will disable it. Plus it's disabled by default in many programming languages. For entities I can decode them, for other weird stuff I use regex to filter them out. And this is not the kind of thing to argue, because 1.it won't stop and 2.it works for me. – quantum
46 upvote
  flag
@xiaomao indeed, if having to know all the gotchas and workarounds to get an 80% solution that fails the rest of the time "works for you", I can't stop you. Meanwhile, I'm over on my side of the fence using parsers that work on 100% of syntactically valid XML. – Charles Duffy
281 upvote
  flag
I once had to pull some data off ~10k pages, all with the same HTML template. They were littered with HTML errors that caused parsers to choke, and all their styling was inline or with <font> etc.: no classes or IDs to help navigate the DOM. After fighting all day with the "right" approach, I finally switched to a regex solution and had it working in an hour. – Paul A Jungwirth
8 upvote
  flag
you can use tidy to create valid XHTML – Sebastian
2 upvote
  flag
Using regex on a large page to get all the data in a large table is much faster than using xpath.. – Izzy
25 upvote
  flag
@CharlesDuffy: definitely less fragile. When the third-party changes their html, they are much more likely to change the structure (breaking your xpaths) than the leaf nodes you are scraping. Scraping is not parsing. Scraping is pulling specific bits of data from a puddle of designer contaminated crap you don't care about. You DO NOT WANT to parse that puddle. You want to do only the absolute minimum amount of "parsing" that will get you your data. You don't CARE about the structure. You care about your bits of data. – Michael Johnston
5 upvote
  flag
@CharlesDuffy you're talking about XML parser, since when an XML parser can read a HTML page? Have you ever tried to load a random webpage from the Internet via an XmlParser (class in an arbitrary language)? I would take it to the level, that probably it's possible that even some W3C valid XHTMLs would fail using XML parsers... look at this page's source for example meta name="twitter:card" is not closed, neither the <link>s. – TWiStErRob
1 upvote
  flag
@SebastianGodelet which will be impossible to parse if the original was non-compliant. – kinokijuf
4 upvote
  flag
The key point for me is that XML parsers can't parse actual web pages found in the wild. A single <p> with no matching </p> will crash all the XML parsers I've used. – Phil Goetz
5 upvote
  flag
I'm sure whoever has to maintain or extend your regexes wished you used a proper xml parser, regardless of the speed. It's not that much slower and I guarantee the code will tend to be much more readable and extensible. – MattNewton
upvote
  flag
There is this with regard to regex w3.org/TR/CSS2/grammar.html – Robert Rocha
4 upvote
  flag
@PaulAJungwirth OK except that was not really HTML, right? ;) – Alois Mahdal
4 upvote
  flag
A much more sensible way of looking at things. Please can we upvote this one. Yes, a parser is better for a lot of jobs, but then it's not always best. As with most programming, there's never just one right solution. Use your brain. – John Hunt
upvote
  flag
regexes are often less fragile, because xml formats may change , but the semantic nature of the underlying data often does not. xml is, more often than not, used as a proxy for structure, where no real structure exists. – Erik Aronesty
upvote
  flag
actually sometimes regex makes more sense as HTML/XML parsers can't parse code due to errors – zakius
3 upvote
  flag
So its OK to ask Paris Hilton to write a part of an or a limited operating system? – joojaa
upvote
  flag
@CharlesDuffy parsing even some limited subset of HTML is a bad idea indeed, but I guess that Kaitlin used the word parsing by mistake as later he wrote about scraping (extracting I guess) some data from a document (just like the original question). If the data you want extract is a regular language you can extract it from a larger document that is not. in fact context-free parsers use REs to convert a document into a string of lexemes/tokens first. I suppose that data from the original question is regular, but I don't know the structure of xml/html well enough to say for sure. – morgwai
1 upvote
  flag
I'm afraid despite the common rumours, ParisOS will be quite cool. and pink, including rainbowcolored unicorns. – Gewure
upvote
  flag
The first answer is pure FUD and incorrect. RegEx is not limited to handling regular languages only: rexegg.com/regex-vs-regular-expression.html Yes, you should be careful as you should in any parsing/scraping scenario. But if you're afraid of the details, you shouldn't be touching problems like this in the first place. – Erik Reppen
upvote
  flag
@ErikReppen Nope. And forget when HTML elements can be commented out. Trying to work with HTML using regex is categorically a bad idea. – jpmc26
upvote
  flag
@jpmc26 Well you could start by splitting your comments and html strings in script tags out just as you would have to with any parsing/scraping approach required when using an actual HTML DOM rendering parser approach wasn't ideal. Establishing that it requires more than one step to use a tool doesn't invalidate its use. At everybody-else, nor does repeating something that hasn't been true since the '80s, which is that RegEx can only handle "regular language" parsing. – Erik Reppen
upvote
  flag
@ErikReppen I wouldn't recommend anyone who doesn't already know the principles of text analysis try that. Sure, there might be 0.01% of cases where using regex to extract data from HTML might make sense. But if you actually know how to tell the difference, you're not going to be blindly plopping a, "How do I parse HTML with regex?" question on SO. At most, you might run into a specific problem while implementing your own, more efficient parser and should have the skills to explain clearly why just using an existing HTML parser won't work or wasn't efficient enough for you. – jpmc26
upvote
  flag
@ErikReppen Also, writing an HTML parser in pure regex is literally insane. Regular expressions are designed for and work best against regular languages. The extensions allow for modest modifications in clear cut cases that are just beyond normal regular languages, not for developing a complex parser for something as intricate as HTML. – jpmc26
1 upvote
  flag
@jpmc26 RegEx hasn't been limited to regular languages since the mid-'80s. That said, no, probably not an awesome choice for something like an actual web browser parser. – Erik Reppen
upvote
  flag
@ErikReppen I didn't say it was limited to regular languages. I did, however, debunk the notion that the top answer is "pure FUD," because the chance of regular expressions actually working properly against any HTML outside of your direct control is nil. Using a proper HTML parser should be your default, because it's not going to trip over constructs you didn't notice or think of. Only if it's not fast enough for you or something should you even consider a different approach. – jpmc26
upvote
  flag
Also: in perl at least, worst case, you can set the e flag to evaluate. So you could use a proper parser in your regex. – bytepusher
1 upvote
  flag
@jpmc26 But it becomes strawman-ish when you associate all regex/html-related questions with the task of actually writing a web brower's HTML parser. If you are assuming that no HTML can be missed or misread or it's mission-catastrophic-failure, I agree with you. But what % of use cases for questions like this actually apply? I regularly return to this answer because somebody who knows less about the subject matter than I do has linked to it because they're trying to tell me that I can't do what I've already done, which is write link-auditing and inline-css search tools for cms sites. – Erik Reppen
upvote
  flag
@ErikReppen I did no such thing. Instead, I'll flip the tables on you. How many of these questions actually specify that the answer doesn't have to work for arbitrary HTML and then specifically indicate what subset of HTML they can work within? Someone who understands the difficulties that their task entails would be able to do so. As for your tools, if you need speed, write a simpler parser, or if you are willing to accept failure in certain cases, sure, use regex. But I sure hope you tried real HTML tools before you resorted to that. Needlessly giving up reliability is just bad software. – jpmc26

The W3C explains parsing in a pseudo regexp form:
W3C Link

Follow the var links for QName, S, and Attribute to get a clearer picture.
Based on that you can create a pretty good regexp to handle things like stripping tags.

2 upvote
  flag
That's not a psuedo regexp form, that's an EBNF form, as specified here: XML spec, appendix 6 – Rob G

Disclaimer: use a parser if you have the option. That said...

This is the regex I use (!) to match HTML tags:

<(?:"[^"]*"['"]*|'[^']*'['"]*|[^'">])+>

It may not be perfect, but I ran this code through a lot of HTML. Note that it even catches strange things like <a name="badgenerator"">, which show up on the web.

I guess to make it not match self contained tags, you'd either want to use Kobi's negative look-behind:

<(?:"[^"]*"['"]*|'[^']*'['"]*|[^'">])+(?<!/\s*)>

or just combine if and if not.

To downvoters: This is working code from an actual product. I doubt anyone reading this page will get the impression that it is socially acceptable to use regexes on HTML.

Caveat: I should note that this regex still breaks down in the presence of CDATA blocks, comments, and script and style elements. Good news is, you can get rid of those using a regex...

82 upvote
  flag
I would go with something that works on sane things than weep about not being universally perfect :-) – prajeesh kumar
44 upvote
  flag
Is someone using CDATA inside HTML? – Danubian Sailor
13 upvote
  flag
so you do not actually solve the parsing problem with regexp only but as a part of the parser this may work. PS: working product doesn't mean good code. No offence, but this is how industrial programming works and gets their money – mishmashru
1 upvote
  flag
@ŁukaszL and the comment's upvoters: //allinonescript.com/questions/66837/… – TWiStErRob
19 upvote
  flag
Your regex starts fail on the very shortest possible, valid HTML: <!doctype html><title><</title>. Simple '<!doctype html><title><</title>'.match(/<(?:"[^"]*"['"]*|'[^']*'['"]*|‌​[^'">])+>/g) returns ["<!doctype html>", "<title>", "<</title>"] while should ["<title>", "</title>"]. – Benio
2 upvote
  flag
if we're just trying to match & not match the examples given, /<.([^r>][^>]*)?>/g works :-) // javascript: '<p> <a href="foo"> <br /> <hr class="foo" />'.match(/<.([^r>][^>]*)?>/g) – imma
1 upvote
  flag
How should you put this is a variable in php? $pattern – SuperSpy
34 upvote
  flag
What is a "badge nerator" – Richard de Wit
7 upvote
  flag
To work is what code is made for. Working code is good code by definition. – Florian F
upvote
  flag
There is a difference between pattern matching a fragment of html and pattern matching html while ensuring a valid structure is returned. Tokenising HTML is probably easy with regex but then try dealing with: <a></b> – jgmjgm
1 upvote
  flag
"Is someone using CDATA inside HTML?" - yes, I do. It takes less bytes if you show HTML source code in <pre> tags. – cweiske
1 upvote
  flag
@FlorianF In modern times, good code doesn't just work. It's organized for testing to guide debugging (future environmental adaptations may break your current systems) and scalable to be prepare for new trends. – Vince Emigh
1 upvote
  flag
@VinceEmigh I see what you mean but I somehow disagree. Good code is code that works. Code quality is a good idea insofar as it helps to produce working code that is maintainable. But code quality isn't good per se, it is a means to produce working code that still works in the future. There are plenty of projects with short-lived code where the time is best spent just making it work. – Florian F
2 upvote
  flag
@Florian F That's like saying an old car, burning twice the fuel it should, burning oil, being noisy and hauling only half the weight it should around, is a good car because it works. – user625488
2 upvote
  flag
If efficiency and silence are requirements, then your car doesn't work as required. On the other hand, monster trucks fit your description well and are considered perfectly good cars by their owners. – Florian F
1 upvote
  flag
@Richard-dW it's a bad generator – tbodt
4 upvote
  flag
"Working code is good code by definition" - I completely disagree. This might have been true in the early days of software, but nowadays there's many more requirements to code than just "working". Among others code should be readable/understandable, maintainable, testable and expandable. – Hubert Grzeskowiak
<?php
$selfClosing = explode(',', 'area,base,basefont,br,col,frame,hr,img,input,isindex,link,meta,param,embed');

$html = '
<p><a href="#">foo</a></p>
<hr/>
<br/>
<div>name</div>';

$dom = new DOMDocument();
$dom->loadHTML($html);
$els = $dom->getElementsByTagName('*');
foreach ( $els as $el ) {
    $nodeName = strtolower($el->nodeName);
    if ( !in_array( $nodeName, $selfClosing ) ) {
        var_dump( $nodeName );
    }
}

Output:

string(4) "html"
string(4) "body"
string(1) "p"
string(1) "a"
string(3) "div"

Basically just define the element node names that are self closing, load the whole html string into a DOM library, grab all elements, loop through and filter out ones which aren't self closing and operate on them.

I'm sure you already know by now that you shouldn't use regex for this purpose.

upvote
  flag
If you're dealing with real XHTML then append getElementsByTagName with NS and specify the namespace. – meder omuraliev

It seems to me you're trying to match tags without a "/" at the end. Try this:

<([a-zA-Z][a-zA-Z0-9]*)[^>]*(?<!/)>
8 upvote
  flag
This does not work. For the input '<x a="<b>"/><y>' the matches are x and y, although x is terminated. – ceving

I used a open source tool called HTMLParser before. It's designed to parse HTML in various ways and serves the purpose quite well. It can parse HTML as different treenode and you can easily use its API to get attributes out of the node. Check it out and see if this can help you.

If you need this for PHP:

The PHP DOM functions won't work properly unless it is properly formatted XML. No matter how much better their use is for the rest of mankind.

simplehtmldom is good, but I found it a bit buggy, and it is is quite memory heavy [Will crash on large pages.]

I have never used querypath, so can't comment on its usefulness.

Another one to try is my DOMParser which is very light on resources and I've been using happily for a while. Simple to learn & powerful.

For Python and Java, similar links were posted.

For the downvoters - I only wrote my class when the XML parsers proved unable to withstand real use. Religious downvoting just prevents useful answers from being posted - keep things within perspective of the question, please.

I don't know your exact need for this, but if you are also using .NET, couldn't you use Html Agility Pack?

Excerpt:

It is a .NET code library that allows you to parse "out of the web" HTML files. The parser is very tolerant with "real world" malformed HTML.

Whenever I need to quickly extract something from an HTML document, I use Tidy to convert it to XML and then use XPath or XSLT to get what I need. In your case, something like this:

//p/a[@href='foo']

I think the flaw here is that HTML is a Chomsky Type 2 grammar (context free grammar) and RegEx is a Chomsky Type 3 grammar (regular grammar). Since a Type 2 grammar is fundamentally more complex than a Type 3 grammar (see the Chomsky hierarchy), you can't possibly make this work. But many will try, some will claim success and others will find the fault and totally mess you up.

182 upvote
  flag
The OP is asking to parse a very limited subset of XHTML: start tags. What makes (X)HTML a CFG is its potential to have elements between the start and end tags of other elements (as in a grammar rule A -> s A e). (X)HTML does not have this property within a start tag: a start tag cannot contain other start tags. The subset that the OP is trying to parse is not a CFG. – LarsH
83 upvote
  flag
In CS theory, regular languages are a strict subset of context-free languages, but regular expression implementations in mainstream programming languages are more powerful. As noulakaz.net/weblog/2007/03/18/… describes, so-called "regular expressions" can check for prime numbers in unary, which is certainly something that a regular expression from CS theory can't accomplish. – Adam Mihalcin
1 upvote
  flag
@LarsH, that may be strictly true, but only if you can depend on well-formedness. To be able to reliably parse even start tags one would have to accommodate a number of workarounds for (X)HTML syntax errors implemented in real world user agents, and even then probably cross fingers and pray. – eyelidlessness
10 upvote
  flag
@eyelidlessness: the same "only if" applies to all CFGs, does it not? I.e. if the (X)HTML input is not well-formed, not even a full-blown XML parser will work reliably. Maybe if you give examples of the "(X)HTML syntax errors implemented in real world user agents" you're referring to, I'll understand what you're getting at better. – LarsH
56 upvote
  flag
@AdamMihalcin is exactly right. Most extant regex engines are more powerful than Chomsky Type 3 grammars (eg non-greedy matching, backrefs). Some regex engines (such as Perl's) are Turing complete. It's true that even those are poor tools for parsing HTML, but this oft-cited argument is not the reason why. – dubiousjim
21 upvote
  flag
This is the most "full and short" answer here. It leads people to learn basics of formal grammars and languages and hopefully some maths so they will not wast time on hopeless things like solving NP-tasks in polynomial time – mishmashru
upvote
  flag
Look at another point of view here: //allinonescript.com/a/21731443/363573 – Stephan
3 upvote
  flag
You are correct, I just want to drop this: You can write a finite state-machine that is equivalent to any regex expression (finite state-machines are equivalent in power to Type 3 languages.) But to parse HTML you need memory, there is this thing called pushdown automaton that is basically an state-machine with a stack to store values. That can be used to parse HTML, unfortunately it is pretty hard to describe a pushdown automaton in text language (like you can represent state-machines in regex,) it is easier to program it yourself. – Hoffmann
1 upvote
  flag
I used regex for a long time before learning the theory and when I was first told you can't parse things with regex I said that was nonsense which is kind of true, it depends what you mean. Language is ambiguous. You can't parse HTML using only regex. I was using regex but additionally with functions that provided the higher level elements for parsing such as maintaining a stack of the current nesting. I wouldn't try using regex alone for parsing anything complex even if it had added features to make it possible. – jgmjgm
3 upvote
  flag
To say the grammar of language A dictates it's parsing capabilities of another language B based on its grammar, is not valid. For example, just because HTML is Chomsky Type 2 language, doesn't mean you could write pure HTML which could parse any Chomsky Type 3 language. HTML itself is not a language with any features that give it the ability to parse other languages. Please don't say "Javascript", because javascript is not parsed by something written in HTML. – AaronLS
1 upvote
  flag
However, you are on the right track. Since HTML is a Chomsky Type 2 language, then to parse a Chomsky Type 2 language you generally need a stack capability within the parsing language(to track context). Regex doesn't have stack management capability: not because it is a Chomsky Type 3, but simply because the language was not designed with that capability. With some support for some extended constructs like recursion, it is possible in Regex, but would not be easy. Just as HTML lacks any capabilities to parse other languages, Regex lacks a stack capability needed to parse HTML. – AaronLS
upvote
  flag
FWIW HTML is not context-free (although the ways in which it is not are not relevant to this problem). For example, you cannot have unique ids in a context-free grammar. – Tgr
upvote
  flag
RegEx hasn't been limited to regular languages for 30 years now. – Erik Reppen

I recently wrote an HTML sanitizer in Java. It is based on a mixed approach of regular expressions and Java code. Personally I hate regular expressions and its folly (readability, maintainability, etc.), but if you reduce the scope of its applications it may fit your needs. Anyway, my sanitizer uses a white list for HTML tags and a black list for some style attributes.

For your convenience I have set up a playground so you can test if the code matches your requirements: playground and Java code. Your feedback will be appreciated.

There is a small article describing this work on my blog: http://roberto.open-lab.com

upvote
  flag
Your links are dead and this isn't really providing any useful answer. – kenorb

While the answers that you can't parse HTML with regexes are correct, they don't apply here. The OP just wants to parse one HTML tag with regexes, and that is something that can be done with a regular expression.

The suggested regex is wrong, though:

<([a-z]+) *[^/]*?>

If you add something to the regex, by backtracking it can be forced to match silly things like <a >>, [^/] is too permissive. Also note that <space>*[^/]* is redundant, because the [^/]* can also match spaces.

My suggestion would be

<([a-z]+)[^>]*(?<!/)>

Where (?<! ... ) is (in Perl regexes) the negative look-behind. It reads "a <, then a word, then anything that's not a >, the last of which may not be a /, followed by >".

Note that this allows things like <a/ > (just like the original regex), so if you want something more restrictive, you need to build a regex to match attribute pairs separated by spaces.

23 upvote
  flag
+1 for noting that the question is not about parsing full (X)HTML, it's about matching (X)HTML open tags. – LarsH
9 upvote
  flag
Something else most of the answers seem to ignore, is that an HTML parser can very well use regular expressions in its implementation for parts of HTML, and I would be surprised if most parsers didn't do this. – Thayne
upvote
  flag
@Thayne Exactly. When parsing individual tags, a regular expression is the right tool for the job. It is quite ridiculous that one has to scroll halfway down the page to find a reasonable answer. The accepted answer is incorrect because it mixes up lexing and parsing. – kasperd
1 upvote
  flag
The answer given here will fail when an attribute value contains a '>' or '/' character. – Martin L
upvote
  flag
This will work incorrectly on HTML containing comments or CData sections. It will also not work correctly if a quoted attribute contains a > character. I agree what OP suggest can be done with a regex, but the one presented here is far to simplistic. – JacquesB

As many people have already pointed out, HTML is not a regular language which can make it very difficult to parse. My solution to this is to turn it into a regular language using a tidy program and then to use an XML parser to consume the results. There are a lot of good options for this. My program is written using Java with the jtidy library to turn the HTML into XML and then Jaxen to xpath into the result.

upvote
  flag
That was helpful, thanks – Khemraj

Although it's not suitable and effective to use regular expressions for that purpose sometimes regular expressions provide quick solutions for simple match problems and in my view it's not that horrbile to use regular expressions for trivial works.

There is a definitive blog post about matching innermost HTML elements written by Steven Levithan.

In shell, you can parse HTML using:

  • sed though:

    1. Turing.sed
    2. Write HTML parser (homework)
    3. ???
    4. Profit!
  • hxselect from html-xml-utils package

  • vim/ex (which can easily jump between html tags), for example:

    • removing style tag with inner code:

      $ curl -s http://example.com/ | ex -s +'/<style.*/norm nvatd' +%p -cq! /dev/stdin
      
  • grep, for example:

    • extracting outer html of H1:

      $ curl -s http://example.com/ | grep -o '<h1>.*</h1>'
      <h1>Example Domain</h1>
      
    • extracting the body:

      $ curl -s http://example.com/ | tr '\n' ' ' | grep -o '<body>.*</body>'
      <body> <div> <h1>Example Domain</h1> ...
      
  • html2text to plain text parsing:

  • using xpath (XML::XPath perl module), see example here

  • perl or Python (see @Gilles example)

  • for parsing multiple files at once, see: How to parse hundred html source code files in shell?


Related (why you shouldn't use regex match):

12 upvote
  flag
2 upvote
  flag
I’m afraid you did not get the joke, @kenorb. Please, read the question and the accepted answer once more. This is not about HTML parsing tools in general, nor about HTML parsing shell tools, it’s about parsing HTML via regexes. – Palec
upvote
  flag
@Palec I don't get the joke either. Is it nearly impossible to parse HTML with regex? – Abdul
upvote
  flag
No, @Abdul. It is completely, provably (in the mathematical sense) impossible. – Palec
upvote
  flag
@Palec Is that mathematical sense in relation to VladGudim's answer on the grammar types? Or something else? – Abdul
2 upvote
  flag
Yes, that answer summarizes it well, @Abdul. Note that, however, regex implementations are not really regular expressions in the mathematical sense -- they have constructs that make them stronger, often Turing-complete (equivalent to Type 0 grammars). The argument breaks with this fact, but is still somewhat valid in the sense that regexes were never meant to be capable of doing such a job, though. – Palec
2 upvote
  flag
And by the way, the joke I referred to was the content of this answer before kenorb's (radical) edits, specifically revision 4, @Abdul. – Palec

This may do:

<.*?[^/]>

Or without the ending tags:

<[^/].*?[^/]>

What's with the flame wars on HTML parsers? HTML parsers must parse (and rebuild!) the entire document before it can categorize your search. Regular expressions may be a faster / elegant in certain circumstances. My 2 cents...

There are some nice regexes for replacing HTML with BBCode here. For all you nay-sayers, note that he's not trying to fully parse HTML, just to sanitize it. He can probably afford to kill off tags that his simple "parser" can't understand.

For example:

$store =~ s/http:/http:\/\//gi;
$store =~ s/https:/https:\/\//gi;
$baseurl = $store;

if (!$query->param("ascii")) {
    $html =~ s/\s\s+/\n/gi;
    $html =~ s/<pre(.*?)>(.*?)<\/pre>/\[code]$2\[\/code]/sgmi;
}

$html =~ s/\n//gi;
$html =~ s/\r\r//gi;
$html =~ s/$baseurl//gi;
$html =~ s/<h[1-7](.*?)>(.*?)<\/h[1-7]>/\n\[b]$2\[\/b]\n/sgmi;
$html =~ s/<p>/\n\n/gi;
$html =~ s/<br(.*?)>/\n/gi;
$html =~ s/<textarea(.*?)>(.*?)<\/textarea>/\[code]$2\[\/code]/sgmi;
$html =~ s/<b>(.*?)<\/b>/\[b]$1\[\/b]/gi;
$html =~ s/<i>(.*?)<\/i>/\[i]$1\[\/i]/gi;
$html =~ s/<u>(.*?)<\/u>/\[u]$1\[\/u]/gi;
$html =~ s/<em>(.*?)<\/em>/\[i]$1\[\/i]/gi;
$html =~ s/<strong>(.*?)<\/strong>/\[b]$1\[\/b]/gi;
$html =~ s/<cite>(.*?)<\/cite>/\[i]$1\[\/i]/gi;
$html =~ s/<font color="(.*?)">(.*?)<\/font>/\[color=$1]$2\[\/color]/sgmi;
$html =~ s/<font color=(.*?)>(.*?)<\/font>/\[color=$1]$2\[\/color]/sgmi;
$html =~ s/<link(.*?)>//gi;
$html =~ s/<li(.*?)>(.*?)<\/li>/\[\*]$2/gi;
$html =~ s/<ul(.*?)>/\[list]/gi;
$html =~ s/<\/ul>/\[\/list]/gi;
$html =~ s/<div>/\n/gi;
$html =~ s/<\/div>/\n/gi;
$html =~ s/<td(.*?)>/ /gi;
$html =~ s/<tr(.*?)>/\n/gi;

$html =~ s/<img(.*?)src="(.*?)"(.*?)>/\[img]$baseurl\/$2\[\/img]/gi;
$html =~ s/<a(.*?)href="(.*?)"(.*?)>(.*?)<\/a>/\[url=$baseurl\/$2]$4\[\/url]/gi;
$html =~ s/\[url=$baseurl\/http:\/\/(.*?)](.*?)\[\/url]/\[url=http:\/\/$1]$2\[\/url]/gi;
$html =~ s/\[img]$baseurl\/http:\/\/(.*?)\[\/img]/\[img]http:\/\/$1\[\/img]/gi;

$html =~ s/<head>(.*?)<\/head>//sgmi;
$html =~ s/<object>(.*?)<\/object>//sgmi;
$html =~ s/<script(.*?)>(.*?)<\/script>//sgmi;
$html =~ s/<style(.*?)>(.*?)<\/style>//sgmi;
$html =~ s/<title>(.*?)<\/title>//sgmi;
$html =~ s/<!--(.*?)-->/\n/sgmi;

$html =~ s/\/\//\//gi;
$html =~ s/http:\//http:\/\//gi;
$html =~ s/https:\//https:\/\//gi;

$html =~ s/<(?:[^>'"]*|(['"]).*?\1)*>//gsi;
$html =~ s/\r\r//gi;
$html =~ s/\[img]\//\[img]/gi;
$html =~ s/\[url=\//\[url=/gi;
10 upvote
  flag
Don't do this. Please. – maletor

About the question of the RegExp methods to parse (x)HTML, the answer to all of the ones who spoke about some limits is: you have not been trained enough to rule the force of this powerful weapon, since NOBODY here spoke about recursion.

A RegExp-agnostic colleague notified me this discussion, which is not certainly the first on the web about this old and hot topic.

After reading some posts, the first thing I did was looking for the "?R" string in this thread. The second was to search about "recursion".
No, holy cow, no match found.
Since nobody mentioned the main mechanism a parser is built onto, I was soon aware that nobody got the point.

If an (x)HTML parser needs recursion, a RegExp parser without recursion is not enough for the purpose. It's a simple construct.

The black art of RegExp is hard to master, so maybe there are further possibilities we left out while trying and testing our personal solution to capture the whole web in one hand... Well, I am sure about it :)

Here's the magic pattern:

$pattern = "/<([\w]+)([^>]*?)(([\s]*\/>)|(>((([^<]*?|<\!\-\-.*?\-\->)|(?R))*)<\/\\1[\s]*>))/s";

Just try it.
It's written as a PHP string, so the "s" modifier makes classes include newlines.
Here's a sample note on the PHP manual I wrote on January: Reference

(Take care, in that note I wrongly used the "m" modifier; it should be erased, notwithstanding it is discarded by the RegExp engine, since no ^ or $ anchorage was used).

Now, we could speak about the limits of this method from a more informed point of view:

  1. according to the specific implementation of the RegExp engine, recursion may have a limit in the number of nested patterns parsed, but it depends on the language used
  2. although corrupted (x)HTML does not drive into severe errors, it is not sanitized.

Anyhow it is only a RegExp pattern, but it discloses the possibility to develop of a lot of powerful implementations.
I wrote this pattern to power the recursive descent parser of a template engine I built in my framework, and performances are really great, both in execution times or in memory usage (nothing to do with other template engines which use the same syntax).

33 upvote
  flag
I'll put this in the "Regex which doesn't allow greater-than in attributes" bin. Check it against <input value="is 5 > 3?" /> – Gareth
64 upvote
  flag
If you put something like that in production code, you would likely be shot by the maintainer. A jury would never convict him. – aehiilrs
upvote
  flag
@Gareth: thanks for your objection, but are you sure that putting a greater-than inside an attribute is a valid code? Well, also if not, this evidences another limit to add to the ones I listed above in case to create a greed parser for the real world... But it is not too much to demonstrate the way is not good, do you agree? There are other useful operators in RegExp which allow to check for next occurrences, this should be a proper use for them. – Emanuele Del Grande
1 upvote
  flag
@Bart K.: it is valid only in an HTML 4- document. XHTML documents need the five XML entities encoded. – Emanuele Del Grande
upvote
  flag
If your comments are aimed to nothing but criticize, I see no good results this discussion may reach. – Emanuele Del Grande
3 upvote
  flag
I was the first to say that my solution has some limits, but of course I am available to listen anyone who can help me in improving it. I posted something which costed me time and work, and which results are effective in a number of projects up and running. I thought it could help, proposing the way of a RegExp solution which nobody nearly spoke about (recursion), and which is the only way to parse nested markup patterns (through RegExp, of course). – Emanuele Del Grande
29 upvote
  flag
Regular expressions can't work because by definition they are not recursive. Adding a recursive operator to regular expressions basically makes a CFG only with poorer syntax. Why not use something designed to be recursive in the first place rather than violently insert recursion into something already overflowing with extraneous functionality? – Welbog
3 upvote
  flag
Once again... > is valid pretty much everywhere in XML, and thus in XHTML, see section 2.4 of the XML spec (at xml.com/axml/target.html#syntax for example) – mirod
1 upvote
  flag
You are right, the lesser-than only is not valid inside XML attributes. Thanks to your criticism, I implemented my solution so that it can parse anything inside the attributes :) Beside this, I implemented the parsing of XML prologue, DTDs and CDATA. The only upset is that the mod closed the possibility to answer this discussion for users with less than 10 points, so that I cannot post it. I twitted him the request to unlock it, but had no response. Come to me, enemies, I wait you! :) The more you are, the stronger I become! – Emanuele Del Grande
15 upvote
  flag
My objection isn't one of functionality it is one of time invested. The problem with RegEx is that by the time you post the cutsey little one liners it appears that you did something more efficiently ("See one line of code!"). And of course no one mentions the half hour (or 3) that they spent with their cheat-sheet and (hopefully) testing every possible permutation of input. And once you get past all that when the maintainer goes to figure out or validate the code they can't just look at it and see that it is right. The have to dissect the expression and essentially retest it all over again... – Oorang
14 upvote
  flag
... to know that it is good. And that will happen even with people who are good with regex. And honestly I suspect that overwhelming majority of people won't know it well. So you take one of the most notorious maintenance nightmares and combine it with recursion which is the other maintenance nightmare and I think to myself what I really need on my project is someone a little less clever. The goal is to write code that bad programmers can maintain without breaking the code base. I know it galls to code to the least common denominator. But hiring excellent talent is hard, and you often... – Oorang
upvote
  flag
Chuck Norris would be able to parse all of you! – Jan

Here is a PHP based parser that parses HTML using some ungodly regex. As the author of this project, I can tell you it is possible to parse HTML with regex, but not efficient. If you need a server-side solution (as I did for my wp-Typography WordPress plugin), this works.

1 upvote
  flag
htmlawed is another PHP project that parses HTML to filter, convert, etc. Has some nice code if you can figure it out! – user594694
upvote
  flag
No you can’t parse HTML with regex. But for some subsets, it may work. – mirabilos

If you only want the tag names it should be possible to do this via regex.

<([a-zA-Z]+)(?:[^>]*[^/] *)?> 

should do what you need. But I think the solution of "moritz" is already fine. I didn't see it in the beginning.

For all downvoters: In some cases it just makes sense to use regex, because it can be the easiest and quickest solution. I agree that in general you should not parse HTML with regex. But regex can be a very powerful tool when you have a subset of HTML where you know the format and you just want to extract some values. I did that hundreds of times and almost always achieved what I wanted.

The OP doesn't seem to say what he needs to do with the tags. For example, does he need to extract inner text, or just examine the tags?

I'm firmly in the camp that says RegEx is not the be-all, end-all text parser. I've written a large amount of text-parsing code including this code to parse HTML tags.

While it's true I'm not all that great with RegEx, I consider regular expressions just too rigid and hard to maintain for this sort of parsing.

There are people that will tell you that the Earth is round (or perhaps that the Earth is an oblate spheroid if they want to use strange words). They are lying.

There are people that will tell you that Regular Expressions shouldn't be recursive. They are limiting you. They need to subjugate you, and they do it by keeping you in ignorance.

You can live in their reality or take the red pill.

Like Lord Marshal (is he a relative of the Marshal .NET class?), I have seen the Underverse Stack Based Regex-Verse and returned with powers knowledge you can't imagine. Yes, I think there were an Old One or two protecting them, but they were watching football on the TV, so it wasn't difficult.

I think the XML case is quite simple. The RegEx (in the .NET syntax), deflated and coded in base64 to make it easier to comprehend by your feeble mind, should be something like this:

7L0HYBxJliUmL23Ke39K9UrX4HShCIBgEyTYkEAQ7MGIzeaS7B1pRyMpqyqBymVWZV1mFkDM7Z28
995777333nvvvfe6O51OJ/ff/z9cZmQBbPbOStrJniGAqsgfP358Hz8itn6Po9/3eIue3+Px7/3F
86enJ8+/fHn64ujx7/t7vFuUd/Dx65fHJ6dHW9/7fd/t7fy+73Ye0v+f0v+Pv//JnTvureM3b169
OP7i9Ogyr5uiWt746u+BBqc/8dXx86PP7tzU9mfQ9tWrL18d3UGnW/z7nZ9htH/y9NXrsy9fvPjq
i5/46ss3p4z+x3e8b452f9/x93a2HxIkH44PpgeFyPD6lMAEHUdbcn8ffTP9fdTrz/8rBPCe05Iv
p9WsWF788Obl9MXJl0/PXnwONLozY747+t7x9k9l2z/4vv4kqo1//993+/vf2kC5HtwNcxXH4aOf
LRw2z9/v8WEz2LTZcpaV1TL/4c3h66ex2Xv95vjF0+PnX744PbrOm59ZVhso5UHYME/dfj768H7e
Yy5uQUydDAH9+/4eR11wHbqdfPnFF6cv3ogq/V23t++4z4620A13cSzd7O1s/77rpw+ePft916c7
O/jj2bNnT7e/t/397//M9+ibA/7s6ZNnz76PP0/kT2rz/Ts/s/0NArvziYxVEZWxbm93xsrUfnlm
rASN7Hf93u/97vvf+2Lx/e89L7+/FSXiz4Bkd/hF5mVq9Yik7fcncft9350QCu+efkr/P6BfntEv
z+iX9c4eBrFz7wEwpB9P+d9n9MfuM3yzt7Nzss0/nuJfbra3e4BvZFR7z07pj3s7O7uWJM8eCkme
nuCPp88MfW6kDeH7+26PSTX8vu+ePAAiO4LVp4zIPWC1t7O/8/+pMX3rzo2KhL7+8s23T1/RhP0e
vyvm8HbsdmPXYDVhtpdnAzJ1k1jeufOtUAM8ffP06Zcnb36fl6dPXh2f/F6nRvruyHfMd9rgJp0Y
gvsRx/6/ZUzfCtX4e5hTndGzp5jQo9e/z+s3p1/czAUMlts+P3tz+uo4tISd745uJxvb3/v4ZlWs
mrjfd9SG/swGPD/6+nh+9MF4brTBRmh1Tl5+9eT52ckt5oR0xldPzp7GR8pfuXf5PWJv4nJIwvbH
W3c+GY3vPvrs9zj8Xb/147/n7/b7/+52DD2gsSH8zGDvH9+i9/fu/PftTfTXYf5hB+9H7P1BeG52
MTtu4S2cTAjDizevv3ry+vSNb8N+3+/1po2anj4/hZsGt3TY4GmjYbEKDJ62/pHB+3/LmL62wdsU
1J18+eINzTJr3dMvXr75fX7m+MXvY9XxF2e/9+nTgPu2bgwh5U0f7u/74y9Pnh6/OX4PlA2UlwTn
xenJG8L996VhbP3++PCrV68QkrjveITxr2TIt+lL+f3k22fPn/6I6f/fMqZvqXN/K4Xps6sazUGZ
GeQlar49xEvajzI35VRevDl78/sc/b7f6jkG8Va/x52N4L9lBe/kZSh1hr9fPj19+ebbR4AifyuY
12efv5CgGh9TroR6Pj2l748iYxYgN8Z7pr0HzRLg66FnRvcjUft/45i+pRP08vTV6TOe2N/9jv37
R9P0/5YxbXQDeK5E9R12XdDA/4zop+/9Ht/65PtsDVlBBUqko986WsDoWqvbPD2gH/T01DAC1NVn
3/uZ0feZ+T77fd/GVMkA4KjeMcg6RcvQLRl8HyPaWVStdv17PwHV0bOB9xUh7rfMp5Zu3icBJp25
D6f0NhayHyfI3HXHY6YYCw7Pz17fEFhQKzS6ZWChrX+kUf7fMqavHViEPPKjCf1/y5hukcyPTvjP
mHQCppRDN4nbVFPaT8+ekpV5/TP8g/79mVPo77PT1/LL7/MzL7548+XvdfritflFY00fxIsvSQPS
mvctdYZpbt7vxKRfj3018OvC/hEf/79lTBvM3debWj+b8KO0wP+3OeM2aYHumuCAGonmCrxw9cVX
X1C2d4P+uSU7eoBUMzI3/f9udjbYl/el04dI7s8fan8dWRjm6gFx+NrKeFP+WX0CxBdPT58df/X8
DaWLX53+xFdnr06f/szv++NnX7x8fnb6NAhIwsbPkPS7iSUQAFETvP2Tx8+/Og0Xt/yBvDn9vd/c
etno8S+81QKXptq/ffzKZFZ+4e/743e8zxino+8RX37/k595h5/H28+y7fPv490hQdJ349E+txB3
zPZ5J/jsR8bs/y1j2hh/2fkayOqEmYcej0cXUWMN7QrqBwjDrVZRfyQM3xjj/EgYvo4wfLTZrnVS
ebdKq0XSZJvzajKQDUv1/P3NwbEP7cN5+Odivv9/ysPfhHfkOP6b9Fl+91v7LD9aCvp/+Zi+7lLQ
j0zwNzYFP+/Y6r1NcFeDbfBIo8rug3zS3/3WPumPlN3/y8f0I2X3cz4FP+/Y6htSdr2I42fEuSPX
/ewpL4e9/n1evzn94hb+Plpw2+dnbyh79zx0CsPvbq0lb+UQ/h7xvqPq/Gc24PnR18fzVrp8I57d
mehj7ebk5VdPnp+d3GJOSP189eTsaXyk/JV7l98j4SAZgRxtf7x155PR+O6jz36Pw9/1Wz/+e/5u
v//vbsfQAxobws8M9v7xLXp/785/395ED4nO1wx5fsTeH4LnRva+eYY8rpZUBFb/j/jfm8XAvfEj
4/b/ljF1F9B/jx5PhAkp1nu/+y3n+kdZp/93jWmjJ/M11TG++VEG6puZn593PPejoOyHMQU/79jq
GwrKfpSB+tmcwZ93XPkjZffDmIKfd2z1DSm7bmCoPPmjBNT74XkrVf71I/Sf6wTU7XJA4RB+lIC6
mW1+xN5GWw1/683C5rnj/m364cmr45Pf6/SN9H4Us4LISn355vjN2ZcvtDGT6fHvapJcMISmxc0K
MAD4IyP6/5Yx/SwkP360FvD1VTH191mURr/HUY+2P3I9boPnz7Ju/pHrcWPnP3I9/r/L3sN0v52z
0fEgNrgbL8/Evfh9fw/q5Xf93u/97vvf+2Lx/e89L7+/Fe3iZ37f34P5h178kTfx/5YxfUs8vY26
7/d4/OWbb5++ogn7PX5XzOHtOP3GrsHmqobOVO/8Hh1Gk/TPl198QS6w+rLb23fcZ0fMaTfjsv29
7Zul7me2v0FgRoYVURnf9nZEkDD+H2VDf8hjeq8xff1s6GbButNLacEtefHm9VdPXp++CRTw7/v9
r6vW8b9eJ0+/PIHzs1HHdyKE/x9L4Y+s2f+PJPX/1dbsJn3wrY6wiqv85vjVm9Pnp+DgN8efM5va
j794+eb36Xz3mAf5+58+f3r68s230dRvJcxKn/l//oh3f+7H9K2O0r05PXf85s2rH83f/1vGdAvd
w+qBFqsoWvzspozD77EpXYeZ7yzdfxy0ec+l+8e/8FbR84+Wd78xbvn/qQQMz/J7L++GPB7N0MQa
2vTMBwjDrVI0PxKGb4xxfiQMX0cYPuq/Fbx2C1sU8yEF+F34iNsx1xOGa9t6l/yX70uqmxu+qBGm
AxlxWwVS11O97ULqlsFIUvUnT4/fHIuL//3f9/t9J39Y9m8W/Tuc296yUeX/b0PiHwUeP1801Y8C
j/9vz9+PAo8f+Vq35Jb/n0rAz7Kv9aPA40fC8P+RMf3sC8PP08DjR1L3DXHoj6SuIz/CCghZNZb8
fb/Hf/2+37tjvuBY9vu3jmRvxNeGgQAuaAF6Pwj8/+e66M8/7rwpRNj6uVwXZRl52k0n3FVl95Q+
+fz0KSu73/dtkGDYdvZgSP5uskadrtViRKyal2IKAiQfiW+FI+tET/9/Txj9SFf8SFf8rOuKzagx
+r/vD34mUADO1P4/AQAA//8=

The options to set is RegexOptions.ExplicitCapture. The capture group you are looking for is ELEMENTNAME. If the capture group ERROR is not empty then there was a parsing error and the Regex stopped.

If you have problems reconverting it to a human-readable regex, this should help:

static string FromBase64(string str)
{
    byte[] byteArray = Convert.FromBase64String(str);

    using (var msIn = new MemoryStream(byteArray))
    using (var msOut = new MemoryStream()) {
        using (var ds = new DeflateStream(msIn, CompressionMode.Decompress)) {
            ds.CopyTo(msOut);
        }

        return Encoding.UTF8.GetString(msOut.ToArray());
    }
}

If you are unsure, no, I'm NOT kidding (but perhaps I'm lying). It WILL work. I've built tons of unit tests to test it, and I have even used (part of) the conformance tests. It's a tokenizer, not a full-blown parser, so it will only split the XML into its component tokens. It won't parse/integrate DTDs.

Oh... if you want the source code of the regex, with some auxiliary methods:

regex to tokenize an xml or the full plain regex

7 upvote
  flag
@bemace You can try it... In a VM... disconnected from the Internet and from your LAN... Using a 10 foot pole to run it! :-) – xanatos
54 upvote
  flag
Good Lord, it's massive. My biggest question is why? You realize that all modern languages have XML parsers, right? You can do all that in like 3 lines and be sure it'll work. Furthermore, do you also realize that pure regex is provably unable to do certain things? Unless you've created a hybrid regex/imperative code parser, but it doesn't look like you have. Can you compress random data as well? – Justin Morgan
79 upvote
  flag
@Justin I don't need a reason. It could be done (and it wasn't illegal/immoral), so I have done it. There are no limitations to the mind except those we acknowledge (Napoleon Hill)... Modern languages can parse XML? Really? And I thought that THAT was illegal! :-) – xanatos
4 upvote
  flag
Well, we're talking about the theoretical limits of the language; it's not like we just haven't figured out how to do it yet. If you use pure regex, there's always going to be some (X)HTML valid code that breaks it. Maybe it's <foo bar="baz > quux"> or maybe it's a certain nesting depth. Not that I didn't find your post funny (Old Ones - hah!) – Justin Morgan
8 upvote
  flag
Don't try it! If you do, the entire internet will be compressed to a quantum singularity and sucked down the rabbit hole! – Christian Hayter
56 upvote
  flag
Sir, I'm convinced. I'm going to use this code as part of the kernel for my perpetual-motion machine--can you believe those fools at the patent office keep rejecting my application? Well, I'll show them. I'll show them all! – Justin Morgan
6 upvote
  flag
@Justin Unless there is a bug, or unless the memory becomes full, I'm pretty sure that any VALID XML can be tokenized. As @John-David Dalton noticed, the specifics of XML are given in Regex-like expressions, so it was quite easy (if not long and harduous) – xanatos
4 upvote
  flag
Does it work for <foo bar="baz > quux">? Because that is in fact valid. So is stuff like <foo bar='"baz" > \'quux\''>. There's a LOT of stuff that looks "wrong" but is still valid XML. – Justin Morgan
4 upvote
  flag
@justin The second one is illegal, you can't escape the quotes with a \ . The first one will work if you close the foo. – xanatos
3 upvote
  flag
@justin And if you don't trust me, //allinonescript.com/questions/1222367/… – xanatos
2 upvote
  flag
Huh, looks like I stand corrected on that one. Fair enough. The basic point is more or less the same, though: Just because you haven't figured out how to break it yet, that doesn't mean it always works. And if it doesn't always work, what's the point? An HTML parser will be more practical AND catch everything. – Justin Morgan
28 upvote
  flag
@Justin So an Xml Parser is by definition bug free, while a Regex isn't? Because if an Xml Parser isn't bug free by definition there could be an xml that make it crash and we are back to step 0. Let say this: both the Xml Parser and this Regex try to be able to parse all the "legal" XML. They CAN parse some "illegal" XML. Bugs could crash both of them. C# XmlReader is surely more tested than this Regex. – xanatos
17 upvote
  flag
No, nothing is bug free: 1) All programs contain at least one bug. 2) All programs contain at least one line of unnecessary source code. 3) By #1 and #2 and using logical induction, it's a simple matter to prove that any program can be reduced to a single line of code with a bug. (from Learning Perl) – sweaver2112
upvote
  flag
(Yes a mutant ant big enough to ride >>is<< biologically impossible. And if it wasn't, it would eat you for breakfast.) – Stephen C
upvote
  flag
+1 this is awesome, @xanatos thanks for this work of art! Out of interest, did you generate the XmlTypes class in your snipt? or was it assembled by hand? – Jacob Stanley
3 upvote
  flag
@JacobStanley No I did them by hand. While the Xml standard is written in pseudo-regex it can't be directly translated. You have to prioritize some parts over others to obtain a parser, so you need the human hand. – xanatos
3 upvote
  flag
It's brillant! I see a useless discussion here because @xanatos seems to prove that is possible to process XML using RegEx and a lot of people say that the approach is useless. I think both are right. I'd like to see some theoretical demonstrations (maybe is a P/NP problem? I have no idea) – jperelli
6 upvote
  flag
The .NET regular language does span XML, and so you can tokenize XML with a .NET regular expression. Tokenization and parsing are very very far from the same. The above regex will accept <A><B><C></B></C></A> (I presume). It is a valid sequence of Xml tokens. This string is in the language of XML tokens. It is not in the language of XML, but that doesn't mean it cannot be tokenized. Just because you cannot parse something doesn't mean that it can't be tokenized. LR(0) parsers can't parse binary strings "00011", but the regex [01]* sure can tokenize them. – Michael Graczyk
upvote
  flag
LR(0) can't parse them because they are left recursive, btw. – Michael Graczyk
2 upvote
  flag
@MichaelGraczyk I remember correctly, the above regex won't accept <A><B><C></B></C></A>. It is a stateful tokenizer, not a stateless tokenizer. – xanatos
3 upvote
  flag
In that case, it isn't a regular expression proper. If it uses backtracking at all, or for whatever reason cannot be expressed in terms of empty strings, literals, and the empty set with concatenations, alternations, Kleene stars thereof (inductive concatenations), then it isn't formally regular. A lot of people here seem to think that all "regular expressions" are true, formal regular expressions. It's true that Thompson NFA regular expressions are not true regular expressions, and are not recursive. However, PCRE regex is not truly regular, and hence can be recursive. – Michael Graczyk
1 upvote
  flag
Of course, you never said that it was a proper regular expression. You just accurately stated that it would tokenize XML. – Michael Graczyk
1 upvote
  flag
Actually, an algorithm CAN be made to compress random data. It would search through all seeds and compare them to the random data to find what seed was used, and find the algorithm used. Then all of that random data can be compressed down to a seed, length, and algorithm. Only problem: it would take days. :-) – uınbɐɥs
5 upvote
  flag
@ShaquinTrifonoff - Then it's not actually random. I think your idea of random data is different from mine. – Justin Morgan
upvote
  flag
@JustinMorgan - You're right. I was thinking of 'normal' random numbers produced by the browser - see Temporary User Tracking In Major Browsers. That document was originally here, but has since been removed. – uınbɐɥs
2 upvote
  flag
This isn't regex, this is full blown parser! While it's perfectly fine to tokenize bits of string by regex, you need some functions around that to make it into a real parser, and that's exactly what this is. By "parsing html with regex", I imagine regex-only-"parser", which is obviously impossible. This is just C# parser that uses regexes every now and then, not regex parser. – enrey
upvote
  flag
huh... I decoded the Base64 using an online decoder... I was expecting more satire - or at least some sarcasm - but all I saw was seemingly random characters... did I do wrong? do I need to compile it or something? or am I disappoint? – Code Jockey
1 upvote
  flag
@CodeJockey It's "deflated and coded in base64", so compressed and then coded... and no satire inside :-) – xanatos
3 upvote
  flag
deflated and coded in base64 to make it easier to comprehend – chiliNUT
4 upvote
  flag
Earth is not an oblate spheroid - it's a geoid, obviously... I thought everybody knows that! – vaxquis
1 upvote
  flag
In this 'verse, there are many nodes that attempt to deceive you with their logic. But there is another verse... a darker verse. In the Regex 'verse, you keep what you ExplicitCapture. – Ber
upvote
  flag
The source link is broken (snipt.net/xanatos/regex-to-tokenize-an-xml) - could you please reupload somewhere? – Peter K
1 upvote
  flag
@PeterK Done: pastebin.com/hzYazFVb (updated the main response). I was lazy so I looked for the original link in the archive.org and then copy-pasted it in pastebin :-) – xanatos
upvote
  flag
Z-Zal-g-g-go? Is that you? – DWoldrich
upvote
  flag
Part of me hopes this is actually real. Part of me prays it's not. You madman. +1 – jpmc26

Don't listen to these guys. You actually can parse context-free grammars with regex if you break the task into smaller pieces. You can generate the correct pattern with a script that does each of these in order:

  1. Solve the Halting Problem.
  2. Square a circle (simulate the "ruler and compass" method for this).
  3. Work out the Traveling Salesman Problem in O(log n). It needs to be fast or the generator will hang.
  4. The pattern will be pretty big, so make sure you have an algorithm that losslessly compresses random data.
  5. Almost there - just divide the whole thing by zero. Easy-peasy.

I haven't figured out the last part yet, but I know I'm getting close. My code keeps throwing CthulhuRlyehWgahnaglFhtagnExceptions lately, so I'm going to port it to VB 6 and use On Error Resume Next. I'll update with the code once I investigate this strange door that just opened in the wall. Hmm.

P.S. Pierre de Fermat also figured out how to do it, but the margin he was writing in wasn't big enough for the code.

63 upvote
  flag
Divison by zero is a much easier problem than the others you mention. If you use intervals, rather than plain floating point arithmetic (which everyone should be but nobody is), you can happily divide something by [an interval containing] zero. The result is simply an interval containing plus and minus infinity. – rjmunro
1 upvote
  flag
@rjmunro I feel that using intervals for arithmetics is something awesome, but I've never seen it before. Do you mean en.wikipedia.org/wiki/Interval_arithmetic ? – whitequark
113 upvote
  flag
Fermat's small margin problem has been solved by soft margins in modern text-editing software. – kd4ttc
2 upvote
  flag
@rjmunro Interval aritmethic (Wikipedia): Division by an interval containing zero is not defined under the basic interval arithmetic. – Luis Masuelli
36 upvote
  flag
Fermat's small margin problem has been solved by Randall Munroe by setting the fontsize to zero: xkcd.com/1381 – heltonbiker
16 upvote
  flag
FYI: Fermat's problem has actually been solved in 1995, and it only took mathematicians 358 years to do so. – jmiserez
4 upvote
  flag
I do all of your steps, except my algorithm for TSP is O(log log n) ,how could I make that O(log n) ? – Lrrr
1 upvote
  flag
It seems pretty clear to me that Fermat had one of the incomplete solutions in mind... Yet another instance of "eye-witness" testimony being unreliable. – Bill IV
3 upvote
  flag
That algorithm is not quite correct. It should contain step 2.5.: Determine the smallest positive integer for which the recursive process of halving (whenever even), or tripling-plus-one (whenever odd), fails to terminate at unity. – Ray Toal
7 upvote
  flag
I was able to bypass that sticky divide-by-zero step by instead using Brownian ratchets yielded from cold fusion...though it only works when I remove the cosmological constant. – Tim Lehner
upvote
  flag
That was helpful, thanks – Khemraj

I like to parse HTML with regular expressions. I don't attempt to parse idiot HTML that is deliberately broken. This code is my main parser (Perl edition):

$_ = join "",<STDIN>; tr/\n\r \t/ /s; s/</\n</g; s/>/>\n/g; s/\n ?\n/\n/g;
s/^ ?\n//s; s/ $//s; print

It's called htmlsplit, splits the HTML into lines, with one tag or chunk of text on each line. The lines can then be processed further with other text tools and scripts, such as grep, sed, Perl, etc. I'm not even joking :) Enjoy.

It is simple enough to rejig my slurp-everything-first Perl script into a nice streaming thing, if you wish to process enormous web pages. But it's not really necessary.

I bet I will get downvoted for this.

HTML Split


Against my expectation this got some upvotes, so I'll suggest some better regular expressions:

/(<.*?>|[^<]+)\s*/g    # get tags and text
/(\w+)="(.*?)"/g       # get attibutes

They are good for XML / XHTML.

With minor variations, it can cope with messy HTML... or convert the HTML -> XHTML first.


The best way to write regular expressions is in the Lex / Yacc style, not as opaque one-liners or commented multi-line monstrosities. I didn't do that here, yet; these ones barely need it.

32 upvote
  flag
"I don't attempt to parse idiot HTML that is deliberately broken." How does your code know the difference? – Kevin Panko
upvote
  flag
Well it doesn't matter much if the HTML is broken or not. The thing will still split HTML into tags and text. The only thing that could foul it up is if people include unescaped < or > characters in text or attributes. In practise, my tiny HTML splitter works well. I don't need an enormous monstrosity chock full of heuristics. Simple solutions are not for everyone...! – Sam Watkins
upvote
  flag
I added some simpler regexps for extracting tags, text, and attributes, for XML / XHTML. – Sam Watkins
upvote
  flag
(get attributes bug 1) /(\w+)="(.*?)"/ assumes double quotes. It will miss values in single quotes. In html version 4 and earlier unquoted value is allowed, if it is a simple word. – David Andersson
upvote
  flag
(get attributes bug 2) /(\w+)="(.*?)"/ may falsely match text that looks like an attribute within an attribute, e.g. <img title="Nope down='up' for aussies" src="..." />. If applied globally, it will also match such things in ordinary text or in html comments. – David Andersson
upvote
  flag
(get attributes bug 3) /(\w+)="(.*?)"/ Optional whitespace should be allowed around the equal sign. – David Andersson
upvote
  flag
(html split bug 1) s/>/>\n/g Since ">" is allowed in data, this may split text lines and confuse subsequent processing. – David Andersson
upvote
  flag
@DavidAndersson, as I said above those regexps are for XML / XHTML, which has a more strict syntax and is easier to parse. If you want a "perfect" parser that can cope with any difficult or malformed HTML input, maybe it's better not to use regexps. But I've found that regexps are more than sufficient for all the html processing tasks I have ever needed to do, and if they don't work for some reason I can tweak them to make them work. In face regexps can be more flexible with some types of broken html that would be rejected by a fussy parser. – Sam Watkins
upvote
  flag
The four bugs above apply to xhtml too. (Except the last part of get attributes bug 1 does not apply to xhtml.) If you put these regexps into production code, would you notice if they occasionally miss input data and/or generate additional fake data? (You have a case for using regexps when trying to parse malformed html, but I think these are too simplistic.) – David Andersson

Here's the solution:

<?php
// here's the pattern:
$pattern = '/<(\w+)(\s+(\w+)\s*\=\s*(\'|")(.*?)\\4\s*)*\s*(\/>|>)/';

// a string to parse:
$string = 'Hello, try clicking <a href="#paragraph">here</a>
    <br/>and check out.<hr />
    <h2>title</h2>
    <a name ="paragraph" rel= "I\'m an anchor"></a>
    Fine, <span title=\'highlight the "punch"\'>thanks<span>.
    <div class = "clear"></div>
    <br>';

// let's get the occurrences:
preg_match_all($pattern, $string, $matches, PREG_PATTERN_ORDER);

// print the result:
print_r($matches[0]);
?>

To test it deeply, I entered in the string auto-closing tags like:

  1. <hr />
  2. <br/>
  3. <br>

I also entered tags with:

  1. one attribute
  2. more than one attribute
  3. attributes which value is bound either into single quotes or into double quotes
  4. attributes containing single quotes when the delimiter is a double quote and vice versa
  5. "unpretty" attributes with a space before the "=" symbol, after it and both before and after it.

Should you find something which does not work in the proof of concept above, I am available in analyzing the code to improve my skills.

<EDIT> I forgot that the question from the user was to avoid the parsing of self-closing tags. In this case the pattern is simpler, turning into this:

$pattern = '/<(\w+)(\s+(\w+)\s*\=\s*(\'|")(.*?)\\4\s*)*\s*>/';

The user @ridgerunner noticed that the pattern does not allow unquoted attributes or attributes with no value. In this case a fine tuning brings us the following pattern:

$pattern = '/<(\w+)(\s+(\w+)(\s*\=\s*(\'|"|)(.*?)\\5\s*)?)*\s*>/';

</EDIT>

Understanding the pattern

If someone is interested in learning more about the pattern, I provide some line:

  1. the first sub-expression (\w+) matches the tag name
  2. the second sub-expression contains the pattern of an attribute. It is composed by:
    1. one or more whitespaces \s+
    2. the name of the attribute (\w+)
    3. zero or more whitespaces \s* (it is possible or not, leaving blanks here)
    4. the "=" symbol
    5. again, zero or more whitespaces
    6. the delimiter of the attribute value, a single or double quote ('|"). In the pattern, the single quote is escaped because it coincides with the PHP string delimiter. This sub-expression is captured with the parentheses so it can be referenced again to parse the closure of the attribute, that's why it is very important.
    7. the value of the attribute, matched by almost anything: (.*?); in this specific syntax, using the greedy match (the question mark after the asterisk) the RegExp engine enables a "look-ahead"-like operator, which matches anything but what follows this sub-expression
    8. here comes the fun: the \4 part is a backreference operator, which refers to a sub-expression defined before in the pattern, in this case, I am referring to the fourth sub-expression, which is the first attribute delimiter found
    9. zero or more whitespaces \s*
    10. the attribute sub-expression ends here, with the specification of zero or more possible occurrences, given by the asterisk.
  3. Then, since a tag may end with a whitespace before the ">" symbol, zero or more whitespaces are matched with the \s* subpattern.
  4. The tag to match may end with a simple ">" symbol, or a possible XHTML closure, which makes use of the slash before it: (/>|>). The slash is, of course, escaped since it coincides with the regular expression delimiter.

Small tip: to better analyze this code it is necessary looking at the source code generated since I did not provide any HTML special characters escaping.

10 upvote
  flag
Does not match valid tags having attributes with no value, i.e. <option selected>. Also does not match valid tags with unquoted attribute values, i.e. <p id=10>. – ridgerunner
1 upvote
  flag
@ridgerunner: Thanks very much for your comment. In that case the pattern must change a bit: $pattern = '/<(\w+)(\s+(\w+)(\s*\=\s*(\'|"|)(.*?)\\5\s*)?)*\s*>/'; I tested it and works in case of non-quoted attributes or attributes with no value. – Emanuele Del Grande
upvote
  flag
How about a space before the tag name: < a href="http://wtf.org" > I'm pretty sure it is legal, but you don't match it. – Floris
5 upvote
  flag
NO sorry, whitespaces before a tagname are illegal. Beyond being "pretty sure" why don't you provide some evidences of your objection? Here are mine, w3.org/TR/xml11/#sec-starttags referred to XML 1.1, and you can find the same for HTML 4, 5 and XHTML, as a W3C validation would also warn if you make a test. As a lot of other blah-blah-poets around here, I did not still receive any intelligent argumentation, apart some hundred of minus to my answers, to demonstrate where my code fails according to the rules of contract specified in the question. I would only welcome them. – Emanuele Del Grande
upvote
  flag
@ridgerunner of course your comment was intelligent and welcome. – Emanuele Del Grande
upvote
  flag
That was helpful, thanks – Khemraj

I agree that the right tool to parse XML and especially HTML is a parser and not a regular expression engine. However, like others have pointed out, sometimes using a regex is quicker, easier, and gets the job done if you know the data format.

Microsoft actually has a section of Best Practices for Regular Expressions in the .NET Framework and specifically talks about Consider[ing] the Input Source.

Regular Expressions do have limitations, but have you considered the following?

The .NET framework is unique when it comes to regular expressions in that it supports Balancing Group Definitions.

For this reason, I believe you CAN parse XML using regular expressions. Note however, that it must be valid XML (browsers are very forgiving of HTML and allow bad XML syntax inside HTML). This is possible since the "Balancing Group Definition" will allow the regular expression engine to act as a PDA.

Quote from article 1 cited above:

.NET Regular Expression Engine

As described above properly balanced constructs cannot be described by a regular expression. However, the .NET regular expression engine provides a few constructs that allow balanced constructs to be recognized.

  • (?<group>) - pushes the captured result on the capture stack with the name group.
  • (?<-group>) - pops the top most capture with the name group off the capture stack.
  • (?(group)yes|no) - matches the yes part if there exists a group with the name group otherwise matches no part.

These constructs allow for a .NET regular expression to emulate a restricted PDA by essentially allowing simple versions of the stack operations: push, pop and empty. The simple operations are pretty much equivalent to increment, decrement and compare to zero respectively. This allows for the .NET regular expression engine to recognize a subset of the context-free languages, in particular the ones that only require a simple counter. This in turn allows for the non-traditional .NET regular expressions to recognize individual properly balanced constructs.

Consider the following regular expression:

(?=<ul\s+id="matchMe"\s+type="square"\s*>)
(?>
   <!-- .*? -->                  |
   <[^>]*/>                      |
   (?<opentag><(?!/)[^>]*[^/]>)  |
   (?<-opentag></[^>]*[^/]>)     |
   [^<>]*
)*
(?(opentag)(?!))

Use the flags:

  • Singleline
  • IgnorePatternWhitespace (not necessary if you collapse regex and remove all whitespace)
  • IgnoreCase (not necessary)

Regular Expression Explained (inline)

(?=<ul\s+id="matchMe"\s+type="square"\s*>) # match start with <ul id="matchMe"...
(?>                                        # atomic group / don't backtrack (faster)
   <!-- .*? -->                 |          # match xml / html comment
   <[^>]*/>                     |          # self closing tag
   (?<opentag><(?!/)[^>]*[^/]>) |          # push opening xml tag
   (?<-opentag></[^>]*[^/]>)    |          # pop closing xml tag
   [^<>]*                                  # something between tags
)*                                         # match as many xml tags as possible
(?(opentag)(?!))                           # ensure no 'opentag' groups are on stack

You can try this at A Better .NET Regular Expression Tester.

I used the sample source of:

<html>
<body>
<div>
   <br />
   <ul id="matchMe" type="square">
      <li>stuff...</li>
      <li>more stuff</li>
      <li>
          <div>
               <span>still more</span>
               <ul>
                    <li>Another &gt;ul&lt;, oh my!</li>
                    <li>...</li>
               </ul>
          </div>
      </li>
   </ul>
</div>
</body>
</html>

This found the match:

   <ul id="matchMe" type="square">
      <li>stuff...</li>
      <li>more stuff</li>
      <li>
          <div>
               <span>still more</span>
               <ul>
                    <li>Another &gt;ul&lt;, oh my!</li>
                    <li>...</li>
               </ul>
          </div>
      </li>
   </ul>

although it actually came out like this:

<ul id="matchMe" type="square">           <li>stuff...</li>           <li>more stuff</li>           <li>               <div>                    <span>still more</span>                    <ul>                         <li>Another &gt;ul&lt;, oh my!</li>                         <li>...</li>                    </ul>               </div>           </li>        </ul>

Lastly, I really enjoyed Jeff Atwood's article: Parsing Html The Cthulhu Way. Funny enough, it cites the answer to this question that currently has over 4k votes.

17 upvote
  flag
System.Text is not part of C#. It's part of .NET. – John Saunders
7 upvote
  flag
In the first line of your regex ((?=<ul\s*id="matchMe"\s*type="square"\s*>) # match start with <ul id="matchMe"...), in between "<ul" and "id" should be \s+, not \s*, unless you want it to match <ulid=... ;) – C0deH4cker
upvote
  flag
@C0deH4cker You are correct, the expression should have \s+ instead of \s*. – Sam
4 upvote
  flag
Not that I really understand it, but I think your regex fails on <img src="images/pic.jpg" /> – Scheintod
3 upvote
  flag
@Scheintod Thank you for the comment. I updated the code. The previous expression failed for self closing tags that had a / somewhere inside which failed for your <img src="images/pic.jpg" /> html. – Sam
<\s*(\w+)[^/>]*>

The parts explained:

<: starting character

\s*: it may have whitespaces before tag name (ugly but possible).

(\w+): tags can contain letters and numbers (h1). Well, \w also matches '_', but it does not hurt I guess. If curious use ([a-zA-Z0-9]+) instead.

[^/>]*: anything except > and / until closing >

>: closing >

UNRELATED

And to fellows who underestimate regular expressions saying they are only as powerful as regular languages:

anbanban which is not regular and not even context free, can be matched with ^(a+)b\1b\1$

Backreferencing FTW!

upvote
  flag
@GlitchMr, that was his point. Modern regular expressions are not technically regular, nor is there any reason for them to be. – alanaktion
3 upvote
  flag
@alanaktion: The "modern" regular expressions (read: with Perl extensions) cannot match within O(MN) (M being regular expression length, N being text length). Backreferences are one of causes of that. The implementation in awk doesn't have backreferences and matches everything within O(MN) time. – xfix

Sun Tzu, an ancient Chinese strategist, general, and philosopher, said:

It is said that if you know your enemies and know yourself, you can win a hundred battles without a single loss. If you only know yourself, but not your opponent, you may win or may lose. If you know neither yourself nor your enemy, you will always endanger yourself.

In this case your enemy is HTML and you are either yourself or regex. You might even be Perl with irregular regex. Know HTML. Know yourself.

I have composed a haiku describing the nature of HTML.

HTML has
complexity exceeding
regular language.

I have also composed a haiku describing the nature of regex in Perl.

The regex you seek
is defined within the phrase
<([a-zA-Z]+)(?:[^>]*[^/]*)?>

I think this might work

<[a-z][^<>]*(?:(?:[^/]\s*)|(?:\s*[^/]))>

And that could be tested here.


As per w3schools...

XML Naming Rules

XML elements must follow these naming rules:

  • Names can contain letters, numbers, and other characters
  • Names cannot start with a number or punctuation character
  • Names cannot start with the letters xml (or XML, or Xml, etc)
  • Names cannot contain spaces
  • Any name can be used, no words are reserved.

And the pattern I used is going to adhere these rules.

60 upvote
  flag
Warning: w3schools should not be treated as an authoritative or reliable reference (ref). Anyway, the rules you listed only apply to the names of elements and attributes; attribute values are much more flexible. You might get away with disallowing > (which is legal but rarely used), but imagine an HREF attribute with no slashes! ;) – Alan Moore
4 upvote
  flag
This expression will work for many element names, however, the XML spec uses letter in the Unicode sense. There are legitimate element names which this won't match. – JamieSee

It's true that when programming it's usually best to use dedicated parsers and APIs instead of regular expressions when dealing with HTML, especially if accuracy is paramount (e.g., if your processing might have security implications). However, I don’t ascribe to a dogmatic view that XML-style markup should never be processed with regular expressions. There are cases when regular expressions are a great tool for the job, such as when making one-time edits in a text editor, fixing broken XML files, or dealing with file formats that look like but aren’t quite XML. There are some issues to be aware of, but they're not insurmountable or even necessarily relevant.

A simple regex like <([^>"']|"[^"]*"|'[^']*')*> is usually good enough, in cases such as those I just mentioned. It's a naive solution, all things considered, but it does correctly allow unencoded > symbols in attribute values. If you're looking for, e.g., a table tag, you could adapt it as </?table\b([^>"']|"[^"]*"|'[^']*')*>.

Just to give a sense of what a more "advanced" HTML regex would look like, the following does a fairly respectable job of emulating real-world browser behavior and the HTML5 parsing algorithm:

</?([A-Za-z][^\s>/]*)(?:=\s*(?:"[^"]*"|'[^']*'|[^\s>]+)|[^>])*(?:>|$)

The following matches a fairly strict definition of XML tags (although it doesn't account for the full set of Unicode characters allowed in XML names):

<(?:([_:A-Z][-.:\w]*)(?:\s+[_:A-Z][-.:\w]*\s*=\s*(?:"[^"]*"|'[^']*'))*\s*/?|/([_:A-Z][-.:\w]*)\s*)>

Granted, these don't account for surrounding context and a few edge cases, but even such things could be dealt with if you really wanted to (e.g., by searching between the matches of another regex).

At the end of the day, use the most appropriate tool for the job, even in the cases when that tool happens to be a regex.

If you're simply trying to find those tags (without ambitions of parsing) try this regular expression:

/<[^/]*?>/g

I wrote it in 30 seconds, and tested here: http://gskinner.com/RegExr/

It matches the types of tags you mentioned, while ignoring the types you said you wanted to ignore.

2 upvote
  flag
I think you mean \/> instead of \\>. – Justin Morgan
upvote
  flag
No, just \> is what I meant; I never meant to edit the regular expression of my original post. – Lonnie Best
2 upvote
  flag
FYI, you don't need to escape angle brackets. Of course, it does no harm to escape them anyway, but look at the confusion you could have avoided. ;) – Alan Moore
upvote
  flag
I sometimes escape unnecessarily when I'm unsure if something is special character or not. I've edited the answer; it works the same but more concise. – Lonnie Best
upvote
  flag
Looking at this now, I don't know why I thought you meant \/, since that would do the exact opposite of the requirements. Maybe I thought you were offering a negative filter pattern. – Justin Morgan
upvote
  flag
That was helpful, thanks – Khemraj

Not the answer you're looking for? Browse other questions tagged or ask your own question.