Libertarians, in general, have a problem when it comes to their advocacy of open borders. Often times, I think that most libertarians believe economics to be the only legitimate form of science and their irrational advocacy of open borders (and free trade) highlights this.
To start, libertarians are staunch defenders of property rights. This is all well and good, except they forget that any government body has property as well. While the anarchist wing will suggest that the State is illegitimate (I mostly agree), they forget that any immigration reform where open borders is made a policy by a State entity will not further the cause of liberty.
This is largely because the immigrants come from countries with different values and different perspectives on the role of government in our lives. Latin America especially has long histories of dictatorships and corrupt democracies that make even our current state of affairs look tame in comparison. To top that off, many of them do not share the same values of liberty and freedom that libertarians believe and this is only further exasperated by the fact that they do not speak English naturally.
The fact that immigrants won't have the same values on liberty and freedom will also serve as a greater frustration when you consider that they or their children will be allowed to vote. This has happened in the past. When German immigrants came to the United States in the 1800s, the idea of government being essential to a child's education and in enforcing morality became much more rampant, for example.
Another problem is that they make the mistake of calling borders "imaginary lines" created by government. Really? So the property line for your land is an imaginary line as well? What you need to understand is that all property that is labeled as "public property" is really just government property. In other words, it is property that is owned by the State and that they are the ones who manage it. And in a representative democracy such as ours, the collective will of the people should dictate how that property is managed. As such, creating an open borders situation is really like having an open property line on your own yard.
The truth is, open borders is not a good policy to advocate right now, especially if your ultimate goal is the elimination or downsizing of government. This is because if the such a thing were to happen, then open borders happens naturally. As it stands right now, between the labor regulations and the welfare state, having immigrants pouring into our country is suicidal and does not spread liberty, but constricts it.