In the post credit scene of Thor (2011), we are shown Loki controlling Selvig as he speaks with Nick Fury about the Tesseract.
In Avengers (2012), Loki makes it to Earth for the first time using the wormhole created by the Tesseract. This is after he's made a deal with the Chitauri.
How is Loki doing that mind control thing on Selvig if he never was on Earth at that time?
This isn't the first time Loki has been to Earth - he shows up to talk to Thor earlier in the film while Thor is incarcerated by SHIELD after trying to retrieve Mjolnir. He's definitely present as he attempts to pick up the hammer, so it's not just an illusion.
As Loki was acting as the King of Asgard at that time, he used the Bifrost to make this trip; however earlier in this film, Loki opens a path to Jotunheim without using the Bifrost to allow the Frost Giants to get into Asgard unseen by Heimdall. In Thor 2, he makes use of a wormhole to transport Thor and Jane from Asgard to Earth and mentions that there are many ways to travel between realms aside from the Bifrost.
We do not know where Loki lands after he lets go of Thor's hand at the end of the film - however, if he can find a way from there to Asgard, he clearly has a route to Earth available.
Worth considering is that he potentially found his way to Thanos who tasked him with retrieving the Tesseract and told him it was on Earth. Thanos could no doubt have provided a method of transport to Loki, although he may not even have needed it. Using the Tesseract to appear on Earth in the Avengers is a bit more theatrical; something Loki is well known for.
As noted in the comment below by Dan Staley, Loki may not have needed to travel to Earth to influence Selvig at all. Note that during the Avengers movie, Loki mentally transports himself to the asteroid where he talks with Thanos and The Other, where he is rebuked for questioning Thanos. After the conversation, he snaps back to wakefulness on Earth. This shows that it is also possible that Loki wasn't on Earth at the point where he was influencing Selvig, just mentally projecting himself instead.